I was a co-founder of CFAR in 2012, same year I got my PhD in math education & left academia. I’d been actively trying to save the world for about a decade at that point. I left CFAR in 2018 due to strong disagreements in approach. I then spent several years focused on a “hippie arc” (meditation, yoga, plant medicine ceremonies, etc.) and “de-mystified” in summer 2024.
Find my non-rationalist writing, social media, and projects at my Linktree.
It seems to me that there are arguments to be made in both directions.[1] It’s not clear to me just yet which stance is correct. Maybe yours is! I don’t know.
My point is that it’s understandable for intelligent people to suspect that there isn’t enough data available yet to produce ASI on the current approach. You might disagree, and maybe your disagreement is even correct, but I don’t think the situation is so vividly clear that it’s incomprehensible why many people wouldn’t be persuaded.
As a quick gesture at the point: as far as I know, all the data LLMs are processing has already gone through a processing filter, namely humans. We produced all the tokens they took in as training data. A newborn, even blind, doesn’t have this limitation — and I’d expect a newborn that was given this limitation somehow very much could have stunted intelligence! I think the analog would be less like a blind newborn and more like a numb one, without tactile or proprioceptive senses.
I think the issue is that the form of the data matters a lot. ChatGPT basically just works with symbols as far as I know. How many symbols are there for it to eat? Are there enough to give the same depth of understanding that a human gets from processing spatial info for instance?
I don’t think this is a hard limit for AI in general. But I can totally understand why there’s skepticism about there being enough digital data right now to produce ASI on the current approach. I’d expect the AIs would need to develop different kinds of data collection, like using robots as sense organs for instance.
Nah. I could have been more precise while spitballing. (For some definition of “could”. I don’t know how to be freeform playful/creative while carefully scrutinizing each detail of what I say for precision and accuracy.) But I meant all that more in the spirit of “Huh, I wonder if evolution did something like this and we’ve been assuming it didn’t. That’d make some sense of why some of our social reform efforts go wonky in these particular ways.”
I don’t know what would be good for us to collectively do. I aspire to have good questions. Good answers would be nice, but I think those mostly fall out of seeking and pondering good questions.
It’s very disconcerting to read “I notice my brain does extra work when I talk with women… wouldn’t it be easier if society were radically altered so that I didn’t have to talk with women?” Like, what? And there’s no way you or anyone else can become more rational about this? This barrier to ideal communication with 50% of people is insurmountable? It’s worth giving up on this one? Hello?
I was not proposing that.
This was fun to look back on a year later.
I like how thoroughly I owned it. How it’s clearly wild speculation that tries to weave together some questionable observations. I also like the questions it raises. I remember how the basic hypothesis — that sexual signaling maybe evolved largely to aim at one’s own sex in the human social context — helped me to notice some interesting questions I hadn’t picked up on before. (E.g., why does slut-shaming seem to more target women who doll up for the male gaze instead of women who are actually pretty openly DTF when there’s a difference? Is my perception off, or is this really a trend, and if so then why?)
I notice that a year later I’m simply unconvinced of the basic idea. I don’t know why. It might be that there’s something true that (some part of me thinks) works best if it goes unrecognized, so I need to blind myself to it. Or, maybe it’s just malarky. I don’t know!
The main thing, I think, is that on the whole the framework doesn’t feel wholesome. It’s not totally icky to me. Just… off? Like, sure, respecting people and accounting for sexual dynamics is important. There are probably built-in tendencies along these lines. Those tendencies seem interesting to examine and explore. But despite my attempt to be descriptive rather than normative in this post, there’s a kind of implication that trying to be attractive to the opposite sex is defecting on culture. That implication seems kind of weird to me now.
To the extent that there’s a glitchiness in trying to be attractive to others, I now suspect it’s more to do with Goodhart drift. A guy who’s just living his life and thus ends up attractive is less likely to be creating a hollow illusion than some other guy who chooses what to do based on what he thinks will get him a girl. As just one example type! It’s a very general point about trying to affect how people see you, instead of just doing stuff and letting people see the truth: the former tends to invoke Goodhart, and thus people are often distrustful of it. But that doesn’t mean it doesn’t work sometimes anyway! Or even that it’s bad.
The bonobo analogy was unfortunate. It was a good illustration, except that it was confusing. I mention the problem in one of the comments. The fact that bonobos specifically use sex is irrelevant to why I named them. The point was that when there’s a potentially scarce resource introduced, the tribe starts by affirming their within-tribe bonds instead of leaping into competition for the resource. (Or at least so I’m told!) The fact that they do so with sex wasn’t relevant to the analogy. Since the rest of the post was about sexual dynamics, I think that particular choice of analogy was unnecessarily confusing.
All that said: I like the kind of wild-yet-owned speculation this post is made of. It’s fun and thought-provoking. I find it stimulating just to reread!
I’d love to see something that massages the various observations here into a better explanation. In particular, there’s nothing meaty here that really tackles why there’s an asymmetry between men and women in terms of intrasex popularity. (Women swoon over men who are (a) beloved and trusted by other men and/or (b) highly desired by other women. Men don’t seem to care whether a woman has lots of girlfriends who love her and are all over the place when it comes to lots of other men wanting her.) The loose model doesn’t predict it and can’t even neatly explain it. Or at least I don’t see how!
I honestly don’t know how much, if any, further thinking or insight or experimentation arose from this post. It might have just amounted to a passing entertainment. It affected some of my thinking and actions in the first half of this year, but I don’t think the ideas have meaningfully come to mind since roughly the summer.
I’ve changed how I want to talk about all this stuff quite a bit since 2018. I don’t talk about “Looking” for the most part anymore. Not for carefully thought-out reasons. I just don’t like the feel of trying to describe this stuff that way anymore. It feels over-reified and too… prideful. Not just about me, but as in, the framework seems to imply a skill or capacity a person has or doesn’t have, and that it’s better if they have it. I no longer think that’s how grace works.
With that caveat: yes probably? I’m guessing Focusing can help. I’m not sure though!
My impression these days is that the kenshō “insight” is basically what you recognize when you stop restricting your perceptions with your cognitive frames. The tricky part is that what I just said is a cognitive frame, so the conceptual mind can take what I just said and claim to have some kind of understanding of the kenshō thing. But it can’t. It literally cannot understand it. It’s like an LLM with a pure text interface talking about truly appreciating visual art: it might give some amazing and even helpful analysis, but it doesn’t have the right type of input or processing to see a painting at all.
Focusing might help by giving the system a way of orienting to things in a non-conceptual way. The conceptual mind can still create frames during and after, but they’re only kind of helpful to the Focusing process, so the conceptual mind can’t lead the Focusing effort.
“You are not the king of your brain. You are the creepy guy standing next to the king going, ‘A most judicious choice, sire’.”
(Here I’m hinting at viewing “you” as the identity structure that lives within the conceptual mind — what some spiritual/mystical places sometimes mean when they say “ego”.)
So, that’s my guess. In short, I’d guess yes? But I really don’t know.
You ask a good question. I have a lot of thoughts about it. Different answers at different levels. Like, what should a civilization do vs. what should a parent do vs. what should a teacher do? Different answers.
The overall theme, though, is to remove coercion and appeal to native fascination instead. If you have something of value to the student to offer, then in practice there’s a way to either (a) show the student that value or (b) earn the student’s trust that you’re tracking what they care about such that when you say “Trust me” they know there’s something good there even if they can’t see it for themselves just yet.
If you’re aiming to be a teacher… well, it’s tricky because last I checked, the systems you’re embedded in impose mandatory coercion. You have to cover certain topics, often in a certain order, within a certain window of time, etc. Especially since “No Child Left Behind” tied funding to test scores. And parents get mad and start rattling sabres if their kids come back from math class with a bunch of weird stuff the parents don’t recognize. Although maybe that was just the Boomers.
But that said! There are clever ways of working within these social constraints. If you can do that, the overall thrust for a teacher is to prioritize being curious about how the students are thinking rather than on getting them to understand certain concepts.
The lion’s share of work for a really good math teacher is in identifying zinger questions. You have to see how a student is thinking about a problem, and follow their contours of reasoning, and notice where it’s going to run them into trouble. You could just tell them about the trouble, but it’s far more effective to ask them to explain something or figure out something that will lead them right to the paradox spot.
After a while you’ll probably develop a really rich repertoire of such questions. And maybe more preciously, you’ll be familiar with a vast library of thinking styles that students actually use in the parts of math that you teach. This is what the education literature refers to as “pedagogical content knowledge” or “PCK” (which is where the CFAR class on “Seeking PCK” came from).
That’s my main answer. Two other points worth mentioning:
By the time they’re in high school, the basis of their math trauma will probably have already formed. You’re not likely to be the tipping point into horror for them.
Math trauma is way more reversible than most people realize.
So don’t worry about that part too much. Just zoom in on what you love about the subject, stay in contact with the kids’ wonder, and aim to be a guide facilitating their exploration of what fascinates them. I think good things follow pretty naturally from that.
Would you say the same of most other class subjects?
I was homeschooled and then studied math education, so I’m not sure. But my passing impression is (a) yes, it applies to most methods of teaching in schools regardless of subject; but (b) math taught this way is particularly damaging.
I want to emphasize that this is my impression. I’m also not entirely sure why math seems to be more damaging. I have guesses. I just observe that e.g. literature hatred or music phobia aren’t nearly as prevalent as math trauma is. Best as I can tell.
I ask because, with the exceptions of reading and persuasive writing, I don’t think that any conventional school subject is more applicable to the average person’s life than grade-school math.
Yes, people can get through life with an astonishing ignorance of mathematics, but they can get through life with an even more astonishing ignorance of social studies, literature, and the sciences.
Well, sure. But people will also pick up the math they need as they need it for the most part. That’s true of most subjects really.
I didn’t learn to read in school. I went to kindergarten before being homeschooled, and they were teaching us the alphabet and some basic words, but I could already read books by then. I learned to read because I wanted to read.
There’s something very weird in our cultural groundwater around what teaching is. It’s like we start with a prescription of subjects and then default to coercion to get students to “know” those subjects. Why? If it’s relevant to their lives, we could learn to point out the connection in a way that feels alive to them. If we can’t do that, then what makes us so sure that it’s relevant for them?
Do you have a different philosophy of education, a different ranking of subjects’ importance, or something else?
Yeah I do. I think the most imporant function of widespread education is to make good citizens. Which is to say, children put through an education system need to come out of it better able to engage with the system that runs their civilization, including the education process for the next generation.
In the United States, I think that puts civics as the most important subject. It’s really key that citizens understand how their government works, what the checks and balances are, how jury nullification works, what forms of corruption actually do arise even within the current system, etc. Otherwise they don’t know how to participate in the government that’s supposedly “by the people, for the people”. This is vastly more important than learning math they don’t naturally pick up in their day-to-day lives.
I think the two things you named are really good though. I wish public education had those as real goals! That’d be nice. I think (1) happens sort of despite the education methods, and (2) happens more through other cultural channels than it does through formal public education. Just my impression.
To me this is exciting. I deduced that the mental architecture you’re describing should be possible. It’s extremely cool to hear someone just name it as a lived experience. Like, what would a mind that’s actually systematically free of Newcomblike self-deception have to be like, assuming the hostile telepaths problem is real? This is one possible solution. Assuming I haven’t misunderstood what you’re describing!
Ah yeah, I think “gaining independence” is a better descriptor of (what I meant by) that solution type.
A few examples:
Framing kids as “disruptive” or “inattentive” or otherwise having the wrong nature if they feel disengaged. This is after informing them what they’re going to study without consulting what’s relevant or interesting to them, and then using social power to require them to study those things. But the problem is supposedly the student, not the system.
Claiming that they’ll need these math tools later in life, and that this justifies adults pressuring the kids to learn those skills now. (This is more bullshit-flavored than gaslight-flavored, but I think they’re psychological neighbors.)
Pretending that because a word problem touches on a topic kids care about, the math is relevant to what the kids like about that topic.
Insisting that forcing kids to take math classes is for their own good, and if the kids don’t see why or don’t agree, then they should believe the adults over their own sense of things.
It makes me so angry. It’s perfectly antithetical to the essence of math as I see it.
In broad strokes I agree with you. Here I was sharing my observation of four cases where a friend was involved this way. One case might have been miscommunication but it doesn’t seem likely to me. The other three definitely weren’t. In one of those I personally knew the guy; I liked him, but he was also emotionally very unstable and definitely not a safe father. I don’t think the abuse was physical in any of those four cases.
! I’m genuinely impressed if you wrote this post without having a mental frame for the concepts drawn from LDT.
Thanks. :)
And thanks for explaining. I’m not sure what “quasi-Kantian” or “quasi-Rawlsian” mean, and I’m not sure which piece of Eliezer’s material you’re gesturing toward, so I think I’m missing some key steps of reasoning.
But on the whole, yeah, I mean defensive power rather than offensive. The offensive stuff is relevant only to the extent that it works for defense. At least that’s how it seems to me! I haven’t thought about it very carefully. But the whole point is, what could make me safe if a hostile telepath discovers a truth in me? The “build power” family of solutions is based on neutralizing the relevance of the “hostile” part.
I think you’re saying something more sophisticated than this. I’m not entirely sure what it is. Like here you say:
Basically, you have to control things orthogonal to your position in the lineup, to robustly improve your algorithm for negotiating with others.
I’m not sure what “the lineup” refers to, so I don’t know what it means for something to be orthogonal to my position in it.
I think I follow and agree with what you’re saying if I just reason in terms of “setting up arms races is bad, all else being equal”.
Or to be more precise, if I take the dangers of adaptive entropy seriously and I view “create adaptive entropy to get ahead” as a confused pseudo-solution. It might be that that’s my LDT-like framework.
I like this way of expressing it. Thanks for sharing.
I think it’s the same core thing I was pointing at in “We’re already in AI takeoff”, only it goes in the opposite direction for metaphors. I was arguing that it’s right to view memes as alive for the same reason we view trees and cats as alive. Grey seems to be arguing to set aside the question and just look at the function. Same intent, opposite approaches.
I think David Deutsch’s article “The Evolution of Culture” is masterful at describing this approach to memetics.
(Though maybe I should say that the therapist needs to either experience unconditional positive regard toward the client, or successfully deceive themselves and the client into thinking that they do. Heh.)
I mean, technically they don’t even need to deceive themselves. They can be consciously judgy as f**k as long as they can mask it effectively. Psychopaths might make for amazing therapists in this one way!
I think the word “power” might be creating some confusion here.
I mean something pretty specific and very practical. I’m not sure how to precisely define it, but here are some examples:
If someone threatens to freak out at you if you disagree with them, and you tend to get overwhelmed and panic when the freak out at you, then they have a kind of power over you. Building power here probably looks like learning to experience them freaking out without you getting overwhelmed.
If someone pays for your rent and food but might stop if they get any hint that you’re gay, it might not be safe to even ask yourself honestly whether you are. You build power here by getting an income, or a source of rent and food, that doesn’t depend on the hostile telepathic benefactor.
If your lover gets turned on by you politically agreeing with them and turned off by disagreement, you might find your political views drifting toward theirs for “unrelated” reasons. One way to build power here is to get other access to sex. Another is to diminish your libido. Another is to break up with them. (Not saying any of these are a great idea. I’m just naming what the solution of “building power” might look like here.)
I’m not familiar with LDT. I can’t comment on that part. Sorry if that means what I just said misses your point.
The fact that Bob has this policy in the first place is more likely when he’s being self-deceptive.
I don’t know if that’s true. It might be. But some possible counterpoints:
People can distrust systems that demand they check. “You have nothing to fear if you have nothing to hide” can get a response of “No” even from people who don’t have anything to hide.
If someone subconsciously thinks they can pull off the illusion of honestly looking while in fact finding nothing, they become more likely to choose to look because they’re self-deceiving.
Someone with a policy of not looking might be better at making their own self-deception unnecessary.
…more often it will be the result of Bob noticing that he’s the sort of person who might have something to hide.
Sure, that way of deciding doesn’t work.
Likewise, if you’re inclined to decide you’re going to dig into possible sources of self-deception because you think it’s unlikely that you have any, then you can’t do this trick.
The hypothetical respect for any self-deception that might be there needs to be unconditional on its existence. Otherwise, for the reason you say, it doesn’t work as well.
(…with some caveats about how people are imperfect telepaths, so some fuzz in implementation here is in practice fine.)
That said, I think you’re right in that if Omega-C is looking only at the choice of whether to look or not, then yes, Omega-C would be right to take the choice as evidence of a deception.
But the whole point is that Omega-C can read what conscious processes you’re using, and can see that you’re deciding for a glomerizing reason.
That’s why why you choose what you do matters so much here. Not just what you choose.
It’s a general rule that if E is strong evidence for X, then ~E is at least weak evidence for ~X.
Conservation of expected evidence is what makes looking relevant. It’s not what makes deciding to look relevant.
If I decide to appease Omega-C by looking, and then I find that I’m self-deceiving, the fact that I chose to look gets filtered. The fact that this is possible is why not finding evidence can matter at all. Otherwise it’d just be a charade.
Relatedly: I have a coin in my pocket. I don’t feel like checking it for bias. Does that make it more likely that the coin is biased? Maybe. But if I could magically show you that I’m not looking because I honestly do not care one way or the other and don’t want to waste the effort, and it doesn’t affect me whether it’s biased or not… then you can’t use my disinterest in checking the coin for bias as evidence of some kind of subconscious deception about the coin’s bias. I’m just refusing to do things that would inform you of the coin’s possible bias.
If this kind of reasoning weren’t possible, then it seems to me that glomerization wouldn’t be possible.
It’s not very hard to detect when someone’s deceiving them self…
A few notes:
Sometimes this is obviously true. I agree.
It’s a curious question why many folk turn their attention away from someone else’s self-deception when it’s obvious. Often they don’t, but sometimes they do. Why they (we) do that is an interesting question worthy of some sincere curiosity.
Confirmation bias. You don’t notice the cases where you don’t pick up on someone else’s self-deception.
…people should notice more and disincentivise that
Boy oh boy do I disagree.
If someone’s only option for dealing with a hostile telepath is self-deception, and then you come in and punish them for using it, thou art a dick.
Like, do you think it helps the abused mothers I named if you punish them somehow for not acknowledging their partners’ abuse? Does it even help the social circle around them?
Even if the “hostile telepath” model is wrong or doesn’t apply in some cases, people self-deceive for some reason. If you don’t dialogue with that reason at all and just create pain and misery for people who use it, you’re making some situation you don’t understand worse.
I agree that getting self-deception out of a culture is a great idea. I want less of it in general.
But we don’t get there by disincentivizing it.
…I went in the other direction: trying to self-deceive little, and instead be self-honest about my real motivations, even if they are “bad PR”.
Yep. I’m not sure why you think this is a “very different” conclusion. I’d say the same thing about myself. The key question is how to handle the cases where becoming conscious of a “bad PR” motivation means it might get exposed.
And you answer that! In part at least. You divide people into three categories based on (a) whether you need occlumency with them at all and (b) whether you need to use occlumency on the fact that you’re using occlumency.
I don’t think of it in terms this explicit, but it’s pretty close to what I do now. People get to see me to the extent that I trust them with what I show them. And that’s conscious.
Am I misunderstanding you somehow?
Moreover, having an extremely difficult high-stakes problem is not just a strong reason to self-deceive less, it’s also strong reason to become more truth-oriented as a community. This means that people with such a common cause should strive to put each other at least in category 2 above, tentatively moving towards 3 (with the caveat of watching out for bad actors trying to exploit that).
I both agree and partly disagree. I tagged your comment with where.
Totally, yes, having a real and meaningful shared problem means we want a truth-seeking community. Strong agreement.
But I think how we “strive” to be truth-seeking might be extremely important. If it’s a virtue instead of an engineering consideration, and if people are shamed or punished for having non-truth-seeking behaviors, then the collective “striving” being talked about will encourage individual self-deception and collective untalkaboutability. It’s an example of inducing adaptive entropy.
Relatedly: mathematicians don’t have truth-seeking collaboration because they’re trying hard to be truth-seeking. They’re trying to solve problems, and they can verify whether their proposed solutions actually solve the problems they’re working on. That means truth-seeking is more useful for what they’re doing than any alternatives are. There’s no need for focusing on the Virtue of Seeking Truth as a culture.
Likewise, there’s no Virtue of Using a Hammer in carpentry.
What puts someone in category 2 or 3 for me isn’t something I can strive for. It’s more like, I can be open to the possibility and be willing to look for how they and I interact. Then I discover how my trust of them shifts. If I try to trust people more than I do, I end up in more adaptive entropic confusion. I’m pretty sure this is lawful on par with thermodynamics.
This might be what you meant. If so, sorry to set up and take a swing at a strawman of what you were saying.
I’ve been reflecting on this since it was posted. Coming back to it from time to time.
I just wanted to make a note saying: received. I believe I see your point, and I’ve been taking it in.
(I also disagree with some of the narrativemancy you employ here about me. It’s difficult for me to publicly agree with anything in your comment here because of a similar mechanism that you’re objecting to in the OP. I wish I didn’t need to add this note. I’d rather just say “I’m hearing something meaningful in what you’re saying and have been taking it in.” That’s the part that matters. But I can do so only if I also register that I very much disagree with your model of what I was trying to do, and in some cases I strongly disagree with your analysis of what I was in fact doing. Thankfully I can still learn from your message anyway. I just wanted to say — more to LW than to you, really — that I’ve heard your objection and have been taking the truth I can find in it seriously.)