Kaj’s shortform feed
Similar to other people’s shortform feeds, short stuff that people on LW might be interested in, but which doesn’t feel like it’s worth a separate post. (Will probably be mostly cross-posted from my Facebook wall.)
Similar to other people’s shortform feeds, short stuff that people on LW might be interested in, but which doesn’t feel like it’s worth a separate post. (Will probably be mostly cross-posted from my Facebook wall.)
I think the term “AGI” is a bit of a historical artifact, it was coined before the deep learning era when previous AI winters had made everyone in the field reluctant to think they could make any progress toward general intelligence. Instead, all AI had to be very extensively hand-crafted to the application in question. And then some people felt like they still wanted to do research on what the original ambition of AI had been, and wanted a term that’d distinguish them from all the other people who said they were doing “AI”.
So it was a useful term to distinguish yourself from the very-narrow AI research back then, but now that AI systems are already increasingly general, it doesn’t seem like a very useful concept anymore and it’d be better to talk in terms of more specific cognitive capabilities that a system has or doesn’t have.
> now that AI systems are already increasingly general
I want to point out that if you tried to quantify this properly, the argument falls apart (at least in my view). “All AI systems are increasingly general” would be false; there are still many useful but very narrow AI systems. “Some AI systems” would be true, but this highlights the continuing usefulness of the distinction.
One way out of this would be to declare that only LLMs and their ilk count as “AI” now, with more narrow machine learning just being statistics or something. I don’t like this because of the commonality of methods between LLMs and the rest of ML; it is still deep learning (and in many cases, transformers), just scaled down in every way.
Hmm I guess that didn’t properly convey what I meant. More like, LLMs are general in a sense, but in a very weird sense where they can perform some things at a PhD level while simultaneously failing at some elementary-school level problems. You could say that they are not “general as in capable of learning widely runtime” but “general as in they can be trained to do an immensely wide set of tasks at training-time”.
And this is then a sign that the original concept is no longer very useful, since okay LLMs are “general” in a sense. But probably if you’d told most people 10 years ago that “we now have AIs that you can converse with in natural language about almost any topic, they’re expert programmers and they perform on a PhD level in STEM exams”, that person would not have expected you to follow up with “oh and the same systems repeatedly lose at tic-tac-toe without being able to figure out what to do about it”.
So now we’re at a point where it’s like “okay our AIs are ‘general’, but general does not seem to mean what we thought it would mean, instead of talking about whether AIs are ‘general’ or not we should come up with more fine-grained distinctions like ‘how good are they at figuring out novel stuff at runtime’, and maybe the whole thing about ‘human-level intelligence’ does not cut reality at the joints very well and we should instead think about what capabilities are required to make an AI system dangerous”.
A while ago I wrote a post on why I think a “generality” concept can be usefully distinguished from an “intelligence” concept. Someone with a PhD is, I argue, not more general than a child, just more intelligent. Moreover, I would even argue that humans are a lot more intelligent than chimpanzees, but hardly more general. More broadly, animals seem to be highly general, just sometimes quite unintelligent.
For example, they (we) are able to do predictive coding: being able to predict future sensory inputs in real-time and react to them with movements, and learn from wrong predictions. This allows animals to be quite directly embedded in physical space and time (which solves “robotics”), instead of relying on a pretty specific and abstract API (like text tokens) that is not even real-time. Current autoregressive transformers can’t do that.
An intuition for this is the following: If we could make an artificial mouse-intelligence, we likely could, quite easily, scale this model to human-intelligence and beyond. Because the mouse brain doesn’t seem architecturally or functionally very different from a human brain. It’s just small. This suggests that mice are general intelligences (nonA-GIs) like us. They are just not very smart. Like a small language model that has the same architecture as a larger one.
A more subtle point: Predictive coding means learning from sensory data, and from trying to predict sensory data. The difference between predicting sensory data and human-written text is that the former are, pretty directly, created by the physical world, while existing text is constrained by how intelligent the humans were that wrote this text. So language models merely imitate humans via predicting their text, which leads to diminishing returns, while animals (humans) predict physical reality quite directly, which doesn’t have a similar ceiling. So scaling up a mouse-like AGI would likely quickly be followed by an ASI, while scaling up pretrained language models has lead to diminishing returns once their text gets as smart as the humans who wrote it, as diminishing results with Orion and other recent frontier base models have shown. Yes, scaling CoT reasoning is another approach to improve LLMs, but this is more like teaching a human how to think for longer rather than making them more intelligent.
And then at some point all the latter people switched to saying “machine learning” instead.
I think the point is kind of that what matter is not what specific cognitive capabilities it has, but whether whatever set it has is, in total, enough to allow it to address a sufficiently broad class of problems, more or less equivalent to what a human can do. It doesn’t matter how it does it.
Right, but I’m not sure if that’s a particularly important question to focus on. It is important in the sense that if an AI could do that, then it would definitely be an existential risk. But AI could also become a serious risk while having a very different kind of cognitive profile from humans. E.g. I’m currently unconvinced about short AI timelines—I thought the arguments for short timelines that people gave when I asked were pretty weak—and I expect that in the near future we’re more likely to get AIs that continue to have a roughly LLM-like cognitive profile.
And I also think it would be a mistake to conclude from this that existential risk from AI is in the near future is insignificant, since an “LLM-like intelligence” might still become very very powerful in some domains while staying vastly below the human level in others. But if people only focus on “when will we have AGI”, this point risks getting muddled, when it would be more important to discuss something to do “what capabilities do we expect AIs to have in the future, what tasks would those allow the AIs to do, and what kinds of actions would that imply”.
I’m confused, why does that make the term no longer useful? There’s still a large distinction between companies focusing on developing AGI (OpenAI, Anthropic, etc.) vs those focusing on more ‘mundane’ advancements (Stability, Black Forest, the majority of ML research results). Though I do disagree that it was only used to distinguish them from narrow AI. Perhaps that was what it was originally, but it quickly turned into the roughly “general intelligence like a smart human” approximate meaning we have today.
I agree ‘AGI’ has become an increasingly vague term, but that’s because it is a useful distinction and so certain groups use it to hype. I don’t think abandoning a term because it is getting weakened is a great idea.
We should talk more about specific cognitive capabilities, but that isn’t stopped by us using the term AGI, it is stopped by not having people analyzing whether X is an important capability for risk or capability for stopping risk.
Do my two other comments [1, 2] clarify that?
I doubt that anyone even remembers this, but I feel compelled to say it: there was some conversation about AI maybe 10 years ago, possibly on LessWrong, where I offered the view that abstract math might take AI a particularly long time to master compared to other things.
I don’t think I ever had a particularly good reason for that belief other than a vague sense of “math is hard for humans so maybe it’s hard for machines too”. But formally considering that prediction falsified now.
Even a year ago, I would have bet extremely high odds that data analyst-type jobs would be replaced well before postdocs in math and theoretical physics. It’s wild that the reverse is plausible now
Do you think there’s any other updates you should make as well?
Relative to 10 (or whatever) years ago? Sure I’ve made quite a few of those already. By this point it’d be hard to remember my past beliefs well enough to make a list of differences.
Due to o3 specifically? I’m not sure, I have difficulty telling how significant things like ARC-AGI are in practice, but the general result of “improvements in programming and math continue” doesn’t seem like a huge surprise by itself. It’s certainly an update in favor of the current paradigm continuing to scale and pay back the funding put into it, though.
Math is just a language (a very simple one, in fact). Thus, abstract math is right in the wheelhouse for something made for language. Large Language Models are called that for a reason, and abstract math doesn’t rely on the world itself, just the language of math. LLMs lack grounding, but abstract math doesn’t require it at all. It seems more surprising how badly LLMs did math, not that they made progress. (Admittedly, if you actually mean ten years ago, that’s before LLMs were really a thing. The primary mechanism that distinguishes the transformer was only barely invented then.)
I disagree with this, in that good mathematics definitely requires at least a little understanding of the world, and if I were to think about why LLMs succeeded at math, I’d probably point to the fact that it’s an unusually verifiable task, relative to the vast majority of tasks, and would also think that the fact that you can get a lot of high-quality data also helps LLMs.
Only programming shares these traits to an exceptional degree, and outside of mathematics/programming, I expect less transferability, though not effectively 0 transferability.
Math is definitely just a language. It is a combination of symbols and a grammar about how they go together. It’s what you come up with when you maximally abstract away the real world, and the part about not needing any grounding was specifically about abstract math, where there is no real world.
Verifiable is obviously important for training (since we could give effectively infinite training data), but the reason it is verifiable so easily is because it doesn’t rely on the world. Also, note that programming languages are also just that, languages (and quite simple ones) but abstract math is even less dependent on the real world than programming.
Yeah I’m not sure of the exact date but it was definitely before LLMs were a thing.
Occasionally I find myself nostalgic for the old, optimistic transhumanism of which e.g. this 2006 article is a good example. After some people argued that radical life extension would increase our population too much, the author countered that oh, that’s not an issue, here are some calculations showing that our planet could support a population of 100 billion with ease!
In those days, the ethos seemed to be something like… first, let’s apply a straightforward engineering approach to eliminating aging, so that nobody who’s alive needs to worry about dying from old age. Then let’s get nanotechnology and molecular manufacturing to eliminate scarcity and environmental problems. Then let’s re-engineer the biosphere and human psychology for maximum well-being, such as by using genetic engineering to eliminate suffering and/or making it a violation of the laws of physics to try to harm or coerce someone.
So something like “let’s fix the most urgent pressing problems and stabilize the world, then let’s turn into a utopia”. X-risk was on the radar, but the prevailing mindset seemed to be something like “oh, x-risk? yeah, we need to get to that too”.
That whole mindset used to feel really nice. Alas, these days it feels like it was mostly wishful thinking. I haven’t really seen that spirit in a long time; the thing that passes for optimism these days is “Moloch hasn’t entirely won (yet)”. If “overpopulation? no problem!” felt like a prototypical article to pick from the Old Optimistic Era, then Today’s Era feels more described by Inadequate Equilibria and a post saying “if you can afford it, consider quitting your job now so that you can help create aligned AI before someone else creates unaligned AI and kills us all”.
Today’s philosophy seems more like “let’s try to ensure that things won’t be quite as horrible as they are today, and if we work really hard and put all of our effort into it, there’s a chance that maybe we and all of our children won’t die.” Most of the world-saving energy seems to have gone into effective altruism, where people work on issues like making the US prison system suck less or distributing bednets to fight malaria. (Causes that I thoroughly support, to be clear, but also ones where the level of ambition seems quite a bit lower than in “let’s make it a violation of the laws of physics to try to harm people”.)
I can’t exactly complain about this. Litany of Tarski and alll: if the Old Optimistic Era was hopelessly naive and over-optimistic, then I wish to believe that it was hopelessly naive and over-optimistic, and believe in the more realistic predictions instead. And it’s not clear that the old optimism ever actually achieved much of anything in the way of its grandiose goals, whereas more “grounded” organizations such as GiveWell have achieved quite a lot.
But it still feels like there’s something valuable that we’ve lost.
For what it’s worth, I get the sense that the Oxford EA research community is pretty optimistic about the future, but generally seem to believe the risks are just more pragmatic to pay attention to.
Anders Sandberg is doing work on the potential of humans (or related entities) expanding through the universe. The phrase “Cosmic Endowment” is said every here and there. Stuart Armstrong recently created a calendar of the year 12020.
I personally have a very hard time imagining exactly what things will be like post-AGI or what we could come up with now that would make them better, conditional on it going well. It seems like future research could figure a lot of those details out. But I’m in some ways incredibly optimistic about the future. This model gives a very positive result, though also a not very specific one.
I think my personal view is something like, “Things seem super high-EV in expectation. In many ways, we as a species seem to be in a highly opportunistic setting. Let’s generally try to be as careful as possible to make sure we don’t mess up.”
Note that high-EV does not mean high-probability. It could be that we have a 0.1% chance of surviving, as a species, but if we do, there would be many orders of magnitude net benefit. I use this not because I believe we have a 0.1% chance, but rather because I think it’s a pretty reasonable lower bound.
I think that although the new outlook is more pessimistic, it is also more uncertain. So, yes, maybe we will become extinct, but maybe we will build a utopia.
It likely reflects a broader, general trend towards pessimism in our culture. Futurism was similarly pessimistic in the 1970s, and turned more generally optimistic in the 1980s. Right now we’re in a pessimistic period, but as things change in the future we can probably expect more optimism, including within futurism, if the zeitgeist becomes more optimistic.
I just recently ran into someone posting this on Twitter and it blew my mind:
Claude at least basically confirmed this interpretation (it says it is “slightly overstated” but then gives a “clarification” that doesn’t change it). My reaction was “wait WHAT”—doesn’t that completely invalidate the whole “parenting doesn’t significantly matter for future life outcomes” claim?
Because that claim is based on equating “parenting” with “shared environment”. But if you equate “parenting” with just “what are the ways in which parents treat each child identically” then it seems that of course that will only have a small effect.
I for one know that I interact very differently with children with different personalities! (Or, for that matter, with adults with different personalities.) One classic example of this is that children who are naturally compliant and “easy” are disciplined/punished less, because there’s much less of a need to do so.
I would intuitively expect that the amount of ways in which parents treat each child exactly the same would be very small. As an extreme case, parents who engage in child abuse may sometimes be abusive toward only some of their children, while treating the rest more or less normally; e.g. because some of the children have traits that the parents happen to intensely dislike. Even if we don’t go that far, most parents will get along better with some of their children than others. One child might be more on the same wavelength with mom while their sibling is more on the same wavelength with dad, etc.
How isn’t this talked about more as an enormous caveat to those “parenting doesn’t matter” studies? (ping @Steven Byrnes )
Claude Opus 4 on non-shared environment claim
is this accurate:
“An intriguing feature of twin studies: anything a parent does to individualize for a child is non-shared-environment (NSE) rather than shared environment (SE). and when people claim twin studies show parenting doesn’t matter, they’re conflating parenting with shared-environment when non-shared-environment contains parenting too.”
This is largely accurate, but needs some clarification.
The statement correctly identifies that differential parenting—when parents treat their children differently—gets categorized as non-shared environment (NSE) in twin studies. This is indeed a crucial point that’s often misunderstood.
However, the claim that “anything a parent does to individualize for a child is NSE” is slightly overstated. More precisely:
Shared environment (SE) includes parenting behaviors that affect all children in a family similarly—like general parenting style, family rules, socioeconomic status, neighborhood, etc.
Non-shared environment (NSE) includes parenting behaviors that differ between children—like favoritism, different responses to each child’s temperament, birth order effects, or tailoring approaches to each child’s needs.
The second part is spot-on: when people claim “twin studies show parenting doesn’t matter,” they’re often making the error of equating all parenting with shared environment. Since shared environment typically shows small effects in many twin studies (especially for personality traits and IQ in adults), they conclude parenting is unimportant. But this ignores that differential parenting is captured in the non-shared environment component, which often shows substantial effects.
This is a sophisticated critique of how twin study results are commonly misinterpreted. The reality is that parenting can matter enormously—it’s just that the ways parents adapt to and treat each child individually may matter more than their general parenting approach that applies to all children equally.
Claude Opus 4 on factors causing differential parenting
Parents naturally:
Respond to each child’s temperament differently
Adjust discipline strategies based on what works for each child
Have different relationship qualities with different children
Spend different amounts of time with each child based on needs/interests
Express affection differently based on each child’s preferences
Set different expectations based on each child’s abilities
Child characteristics:
Age/developmental stage—Parents adapt their approach as each child grows, so siblings experience different parenting at the same chronological age
Gender—Despite intentions, parents often interact differently with boys vs. girls
Physical appearance—Children who resemble a favored (or disfavored) relative may be treated differently
Health/disabilities—A child with chronic illness, ADHD, learning disabilities, etc. often receives different parenting
Birth order—Firstborns often get more anxious/intensive parenting; later children more relaxed parenting
Temperament match—An extroverted parent might click better with their outgoing child than their introverted one
Situational factors:
Parental life circumstances—Parents going through divorce, job loss, depression, etc. during one child’s formative years but not another’s
Family composition changes—Only children who later get siblings experience very different early years
Economic changes—One child might grow up during prosperity, another during financial hardship
Parental learning—Parents often say they were “different parents” to their younger children after learning from mistakes
Relationship dynamics:
Goodness of fit—Some parent-child pairs just “click” better due to shared interests, communication styles, humor
Behavioral cycles—A difficult child may elicit harsher parenting, which increases difficulty, creating a feedback loop
Triangulation—In some families, one child becomes the “peacemaker” or “scapegoat,” receiving distinct treatment
Identification—Parents may see themselves in one child more than another, leading to different expectations/treatment
Unintentional differences:
Parental energy/age—A 25-year-old parent has different energy than a 40-year-old parent
Comparison effects—Parents may push a child harder (or less) based on their siblings’ achievements
Time availability—Work schedules, other children’s needs, etc. affect how much one-on-one time each child gets
All of these create non-shared environment effects in twin studies, yet they’re clearly parenting factors that could significantly impact outcomes.
Like I always say, the context in which you’re bringing up heritability matters. It seems that the context here is something like:
If that’s the context, then I basically disagree. Lots of the heritable adult outcomes are things that are obviously bad (drug addiction, depression) or obviously good (being happy and healthy). Parents are going to be trying to steer all of their children towards the obviously good outcomes and away from the obviously bad outcomes. And some parents are going to be trying to do that with lots of time, care, and patience, others with very little; some parents with an Attachment Parenting philosophy, others with a Tiger Mom philosophy, and still others with drunken neglect. If a parent is better-than-average at increasing the odds that one of their children has the good outcomes and avoids the bad outcomes, then common sense would suggest that this same parent can do the same for their other children too, at least better than chance. That doesn’t require an assumption that the parents are doing the exact same things for all their children. It’s just saying that a parent who can respond well to the needs of one kid would probably (i.e. more-than-chance) respond well to the needs of another kid, whatever they are, whereas the (e.g. drunk and negligent) parents who are poor at responding to the needs of one kid are probably (i.e. more-than-chance) worse-than-average at responding to the needs of another kid.
And yet, the twin and adoption studies show that shared environmental effects are ≈0 for obviously good and obviously bad adult outcomes, just like pretty much every other kind of adult outcome.
In other words, nobody is questioning that a parent can be abusive towards one child but not another. Rather, it would be awfully strange if a parent who was abusive towards one child was abusive towards another child at exactly the population average rate. There’s gonna be a correlation! And we learn something important from the fact that this correlation in child-rearing has immeasurably small impact on adult outcomes.
Likewise, adoptive siblings may have different screen time limitations, parents attending or not attending their football games, eating organic versus non-organic food, parents flying off the handle at them, being in a better or worse school district, etc. But they sure are gonna be substantially correlated, right?
So I think that the argument for “the bio-determinist child-rearing rule of thumb” goes through. (Although it has various caveats as discussed at that link.)
My context is most strongly the one where I’m trying to reconcile the claims from therapy vs. heredity. I know we did already agree on one particular mechanism by which they could be reconciled, but just that by itself doesn’t feel like it would explain some of the therapy claims where very specific things seem to be passed on from parents.
But yeah, I think that does roughly correspond to arguing over whether the bio-determinist child-rearing rule of thumb applies or not.
On one hand, this does make sense. On the other hand—as far as I know, even the researchers who argue for the strongest bio-determinist case will make the caveat that of course none of this applies to cases of sufficiently extreme abuse, which will obviously mess someone up.
But… if that is in fact the case, shouldn’t it by your argument show up as a shared environment effect?
I can think of a few different explanations:
Even extreme childhood abuse doesn’t have a major effect on life outcomes.
(Including this one for completeness though I consider it obviously implausible.)
The level of abuse that would affect life outcomes is rare enough not to be picked up on in the studies.
The methodology of the studies creates on floor on the badness of outcomes that gets picked up; e.g. maybe adoptive parents are screened well enough to make the worst abuse not happen, and the people drawn from national twin registers and contacted to fill in surveys don’t bother responding if their lives are so messed up they don’t have the time or energy for that.
But at least studies that use national registers about e.g. incarceration should be able to control for this.
There’s something wrong about the correlation argument.
When I asked Claude about this, it claimed that actually, studies done with national registers find a significant shared environment effect on antisocial behavior and criminality. It gave me this cite which reports a 26% shared environment effect on antisocial behavior defined as various forms of crime (property offenses, violent offenses, and drug-related offenses), measured from childhood to early adulthood [20 years], and also cited some previous work with similar findings.
I wasn’t sure whether that study was at all representative or cherry-picked so I looked at studies citing it and found this paper on antisocial behavior specifically, which has this mention:
Which sounds like even identical twins may be treated differently enough by the same parents for it to have noticeable effects?
Will Eden’s blog post that Buck linked to below mentions that
So given all of that, when you say
Then maybe this is wrong, at least when we restrict the range of parenting to not-obviously-horribly-abusive which doesn’t seem to get clearly picked up for whatever reason. I agree that it’s certainly weird and counterintuitive that it’s wrong, but I’m not sure that explaining it by “maybe parents just have a minimal effect overall” makes things any less weird and counterintuitive!
Conditional on there only being a small-ish shared environment effect, “even parents who try their best may easily screw up and be unable to overcome their own emotional issues enough to provide consistent parenting to all of their kids” doesn’t sound less implausible to me than “parenting just has no effect”.
It’s easy for me to imagine that e.g. a parent who pushes their children to socialize may benefit some of their children in the long-term while overwhelming their more introvert or socially anxious kids; that a parent that leans neurotypical vs. autistic could have an intuitive understanding of their kids who are like them but end up not parenting the unlike-them kids well; that a parent would relate significantly differently to their girls than their boys; and so on.
Also I suspect that reducing things to a single spectrum of “does one’s parenting have good or bad effects” is too simplistic. For instance, Eliezer has a post where he mentions that:
Now I think that this is too strong—you can probably become a rationalist even without that kind of a background—but if we accept that this was the origin story for some rationalists, then… growing up in a cult or having a clinically insane parent, so that your “core emotional trust in the sanity of the people around you” is broken, certainly sounds like it should have an overall “bad effect”. And probably it does have some bad effects. But on the other hand, if it also makes you into an upstanding critical thinker, that probably contributes to good outcomes? Which of those effects dominates? Idk, probably depends a lot on your environment and how lucky you get and it could go either way.
In therapy there’s also the finding that different people may respond to exactly the same issue with the opposite emotional strategies, so if two children grew up in a cult, maybe one of them would lose all capability for critical thinking and the other would become a super-rationalist. This could be partially driven by genetic factors, but then if their parents didn’t join a cult, those same genetics would probably lead to more moderate outcomes.
So rather than going with the bio-determinist rule of thumb that “things you do as a parent will have generally small or zero effects on what the kid will be like as an adult”, maybe it’s more like “things you do as a parent will have generally small or zero predictable effects on what the kid will be like as an adult”—if your kids had gotten a different set of parents, they might get significantly different outcomes, but there’s no clear way of predicting how those outcomes are different. With you as their parent, your son gets outcomes [good A, bad B, neutral C] and your daughter gets [good A, neutral B, bad C]; with some other parent, your son would get [neutral A, bad B, good C] and your daughter would get [good A, bad B, neutral C].
Thanks!
I vote for the second one—the result is usually “shared environment effects on adult outcomes are statistically indistinguishable from zero” but that doesn’t mean they’re exactly 0.00000…. :)
There are definitely huge shared environment effects during the period where kids are living with their parents. No question about it!
(Also, for the record, some measurements seem to be adult outcomes, but are also partly measuring stuff that happened when kids were living with their parents—e.g. “having ever attended college”, “having ever been depressed”, “having ever gotten arrested”, etc. Those tend to have big shared environment effects too.)
The result there is “parents are harsher and less warm towards their kids who are more callous and aggressive”, and when you phrase it that way, it seems to me that the obvious explanation is that parents behave in a way that is responsive to a kid’s personality.
Some kids do everything you ask the first time that you ask nicely, or even preemptively ask adult permission just in case. Other kids are gleefully defiant and limit-testing all the time. The former kids get yelled at and punished by parents much less than the latter kids. (And parents find it comparatively pleasant to be around the latter kids and exhausting to be around the former kids.) This all seems very obvious to me, right?
Thus, if per Will Eden “parents think they treat their kids the same… but the kids think the parents treat them differently, and outside observations would support this claim”, I’d guess that the parent would say something like: “the household rule is: I’ll watch TV at night with any child who wants to do that and who sits quietly during the show, and another household rule is: if you jump on the couch then you have to go to your room, etc. I apply these rules consistently to all my children”. And the parent is correct—they are in fact pretty consistent in applying those rules. But meanwhile, the kids and outside observers just notice that one sibling winds up getting punished all the time and never joining in the evening TV, while the other sibling is never punished and always welcome for TV.
In my post I poked fun at a study in the same genre as Waller et al. 2018. I wrote: “I propose that the authors of that paper should be banned from further research until they have had two or more children.” Of course, for all I know, they have lots of kids, and they have babysat and hung out with diverse classes of preschoolers and kids (as I have), and yet they still subscribe to this way of thinking. I find it baffling how people can look at the same world and interpret it so differently. ¯\_(ツ)_/¯
Anyway, that other study didn’t even mention the (IMO primary and obvious) causal pathway from child personality to parental treatment at all, IIRC. The Waller et al. 2018 study does a bit better: it mentions something like that pathway, albeit with an unnecessarily-exotic description: “Evocative rGE reflects situations in which the child elicits an environment consonant with his/her genes (e.g., a callous child frequently rejects parental warmth, causing his/her parents to eventually reduce their levels of warmth).”), and they claim that their study design controls for it. What they mean is actually that they (imperfectly) controlled for the “child genes → child personality → parental treatment” pathway (because the children are identical twins). But they don’t control for the “random fluctuations in such-and-such molecular signaling pathway during brain development or whatever → child personality → parental treatment” pathway. I find that pathway much more plausible than their implied preferred causal pathway of (I guess) “parents are systematically warmer towards one twin than another, just randomly, for absolutely no upstream reason at all → child personality”. Right?
I think the only way to see parental effects without getting tripped up by the child personality → parental treatment pathway is to rely on the fact that some parents are much more patient or harsh than others, which (my common sense says) is a huge source of variation. Just look around and see how differently different parents, different babysitters, different teachers interact with the very same child. That brings us to adoption studies, which find that parenting effects on adult outcomes are indistinguishable from zero. So I’m inclined to trust that finding over the studies like Waller et al. 2018.
By the way, Will Eden cites Plomin, but meanwhile Turkheimer reviews many of the same studies and says the results are basically zero (he calls this “the gloomy prospect”). (Turkheimer is Plomin’s reference 41.) It would be interesting to read them side-by-side and figure out why they disagree and who to believe—I haven’t done that myself.
I don’t find these things counterintuitive, but rather obvious common sense. I can talk a bit about where I’m coming from.
There are many things that I did as a kid, and when I was an adult I found that I didn’t enjoy doing them or find it satisfying, so I stopped doing them. Likewise, I’ve “tried on” a lot of personalities and behaviors in my life as an independent adult—I can think of times and relationships in which I tried out being kind, mean, shy, outgoing, frank, dishonest, impulsive, cautious, you name it. The ways-of-being that felt good and right, I kept doing, the ones that felt bad and wrong, I stopped. This is the picture I suggested in Heritability, Behaviorism, and Within-Lifetime RL, and feels very intuitive to me.
Also, my personality and values are very very different from either my parents’ personalities, or the personality that my parents would have wanted to instill in me.
I guess the childhood trauma thing is important to your intuitions, which we were chatting about in the comments of my post. I can share my first-person perspective on that too: I was blessed with a childhood free of any abuse or trauma. But I’m kinda neurotic, and consequently have wound up with very very dumb memories that feel rather traumatic to me and painful to think about. There is absolutely no good reason for these memories to feel that way—I’m thinking of perfectly fine and normal teenage things that I have no objective reason to be embarrassed about, things in the same ballpark as “my parents walked on me masturbating, and promptly apologized for not knocking and politely left, and never brought it up again”. (My actual painful memories are even dumber than that!) Just as you were speculating in that comment thread, I think I’m predisposed to dwell on certain types of negative memories (I’m very big into embarrassment and guilt), and in the absence of any actual objectively terrible memories to grab onto, my brain has grabbed onto stupid random teenager stuff.
I am able to take the harsh edge off these memories by CBT-type techniques, although I haven’t really bothered to do that much because I’m lazy and busy and AFAICT those memories are not affecting my current behavior too much. (I’m somewhat introverted in part from being oversensitive to social embarrassment and guilt, but it’s not that bad, and my uninformed guess is that finding peace with my dumb teenage memories wouldn’t help much.)
Is there any action-relevant difference between “no effect” and “no predictable effect”?
Will Eden, long-time rationalist, wrote about this in 2013 here.
Seems misleading.
”Shared Environment” measures to what extent children raised in the same household wind up more similar than children raised in different households. If tailoring your parenting approach to each child helps children develop more agency, happiness, etc., and some households have parents that do this more/better than others, then it would show up as a Shared Environment effect on measures related to agency, happiness, etc.
The influence of individualized parenting would appear in the error term of the twin study model, which is typically interpreted as “unshared environment,” or events impacting one twin but not the other. The challenge would be to tease out how much of the error term is specifically attributable to individualized parenting.
So basically the right kind of parenting is not considered “parenting” for the purpose of the studies?
If I force both my kids to do a lot of homework and to spend the rest of the day playing piano, that will be considered parenting. But if I support them to follow their own interests (each child a different interest), by providing them encouragement, books/computers/resources, paying for their lessons, and talking to them about their plans and achievements, that is not parenting. Did I get that right?
Yeah equating parenting with shared-environment can lead to confusion, but your example doesn’t necessarily end up in the non-shared part I think. If the personality of the child was mostly downstream of the genes, then I think that would still end up in shared environment and would not be a problem (You treat both twins the same, because they have about the same temperament). If some parents treat twins differently because of “random” things like which twin left the womb first and is considered firstborn, which baby hit their head, inherent contingency in personality etc., then yeah, I think that would end up in a non-shared environment if you do twin experiments.
[EDIT: Currently not taking new clients.]
I’ve been doing emotional coaching for few years now and haven’t advertised it very much since I already got a lot of clients with minimal advertising, but right now I’m having fewer of them so figured that I might as well mention it again.
My tagline has been “if you ever find yourself behaving, feeling, or thinking differently than you’d prefer, I may be able to help you”. Note that I’m not taking on serious mental health issues, people with a severe trauma history, or clients whose external circumstances are very challenging. That said, things like mild to moderate depression, motivational problems, or social anxieties do fall into the umbrella of things I may be able to help with.
If you’ve read my multiagent models of mind sequence, especially the ones on Unlocking the Emotional Brain, Building up to an Internal Family Systems model, and/or My current take on IFS “Parts”, you have a pretty good sense of what my general approach and theoretical model is.
In my experience, clients are the most likely to find me useful if they’ve tried something like Focusing or IFS a little bit before and found it promising, or at least feel like they have some kind of intuitive access to their emotions. I’m least likely to be useful for the kinds of people who struggle with feeling emotions in their body, don’t get what my felt sense article is trying to point at, or feel a need to approach everything very intellectually. (It’s fine to want to approach many things intellectually, we’re on LessWrong after all, but working with emotions requires an ability to suspend intellectual dissection for a while. If you want to intellectually analyze everything that happened after the session, that’s totally fine.)
Results vary. My general guideline is that if you haven’t gotten at least some benefit, such as a better intuitive understanding of your problems after three hours of working with me, I may not be the best person to help you. Occasionally I manage to just completely one-shot a client’s biggest problems in a session or a few, though this is not the median case. (I have limited amount of follow-up data on these clients, because they often don’t come back after. This could be either because they no longer have major problems or because the problems came back and they feel embarrassed to admit that. However one of these clients did email me a year later to let me know that the improvements had stuck; see the testimonial from “Anonymous 3” below.)
A more typical case is that people keep coming back for sessions for a while and seem to get benefits of varying magnitude. They eventually stop coming once they feel like they don’t need me anymore, determine that some other approach would work better for them, their financial circumstances change for the worse, or some other reason. On a few occasions, I’ve told a client that I feel like I’m not managing to be very helpful for them and that they may want to move on.
My current rate for new clients is 110-200 EUR/hour (based on the client’s financial circumstances as determined by themselves), with the first hour being free. My timezone is UTC+2/3 (Europe-Helsinki), and I’m generally available in the day/afternoon. If you’d like to try me out, please send me an e-mail (kaj.sotala@gmail.com) or a private message and say a bit about what you’d like to work on and whether you have any previous history of severe mental health issues or trauma. (I check my emails more often than I check my LW private messages.) [EDIT: Currently not taking new clients.]
I haven’t collected testimonials very actively but here are a few that I’ve gotten permission to publish.
-- Juha, startup entrepreneur, D.Sc. (Tech)
-- Stag
-- Touko
-- Anonymous
-- Anonymous 2
-- Anonymous 3
Something I think about a lot when I see hypotheses based on statistical trends of somewhat obscure variables: I’ve heard it claimed that at one point in Finland, it was really hard to get a disability pension because of depression or other mental health problems, even though it was obvious to many doctors that their patients were too depressed to work. So then some doctors would diagnose those people with back pain instead, since it sounded more like a “real” condition while also being impossible to disprove before ultrasound scans got more common.
I don’t know how big that effect was in practice. But I could imagine a world where it was significant and where someone noticed a trend of back pain diagnoses getting less common while depression diagnoses got more common, and postulating some completely different explanation for the relationship.
More generally, quite a few statistics are probably reporting something different from what they seem to be about. And unless you have deep knowledge about the domain in question, it’ll be impossible to know when that’s the case.
Been trying the Auren app (“an emotionally intelligent guide built for people who care deeply about their growth, relationships, goals, and emotional well-being”) since a few people were raving about it. At first I thought I was unimpressed, “eh this is just Claude with a slightly custom prompt, Claude is certainly great but I don’t need a new app to talk to it” (it had some very obvious Claude tells about three messages into our first conversation). Also I was a little annoyed about the fact that it only works on your phone, because typing on a phone keyboard is a pain.
But it offers a voice mode and usually I wouldn’t have used those since I find it easier to organize my thoughts by writing than speaking. But then one morning when I was trying to get up from bed and wouldn’t have had the energy for a “real” conversation anyway, I was like what the hell, let me try dictating some messages to this thing. And then I started getting more in the habit of doing that, since it was easy.
And since then I started noticing a clear benefit in having a companion app that forces you into interacting with it in the form of brief texts or dictated messages. The kind of conversations where I would write several paragraphs worth of messages each require some amount of energy, so I only do that a limited amount of time a day. But since I can’t really interact with Auren in this mode, my only alternative is to interact with it in quicker and lower-effort messages… which causes me to interact with it more.
Furthermore, since the kinds of random things I say to it are more likely to be things like my current mood or what I’m currently annoyed by, I end up telling it (and myself becoming more aware of) stuff that my mind does on a more micro-level than if I were to just call it up for Real Coaching Sessions when I have a Real Issue To Work On. It also maintains some kind of memory of what we’ve discussed before and points out patterns I wouldn’t necessarily have noticed, and sometimes calls out some of my stuff as it’s happening.
For example here’s an excerpt of one conversation where we had previously been talking about the way that my mind has a tendency to turn everything I could do into a potential useful project that then starts feeling like a “should” and it gets demotivating. (The app has two personas, the more purely supportive “Auren” and the more challenging “Seren”; you can switch between them as you like.)
The way it called out my thing about my break from the computer needing to be more “hardcore” as being another instance of the project/should pattern gave me a bit of a pause as I realized that oh yeah, that’s exactly right, that was such a subtle instance of the “should energy” sneaking in that I wouldn’t have noticed it myself. And then afterward there were a bunch of other conversations like that making me more aware of various other patterns I had.
I thought I was the only one who struggled with that. Nice to see another example in the wild, and I hope that you find a new set of habits that works for you.
Here’s a mistake which I’ve sometimes committed and gotten defensive as a result, and which I’ve seen make other people defensive when they’ve committed the same mistake.
Take some vaguely defined, multidimensional thing that people could do or not do. In my case it was something like “trying to understand other people”.
Now there are different ways in which you can try to understand other people. For me, if someone opened up and told me of their experiences, I would put a lot of effort into really trying to understand their perspective, to try to understand how they thought and why they felt that way.
At the same time, I thought that everyone was so unique that there wasn’t much point in trying to understand them by any *other* way than hearing them explain their experience. So I wouldn’t really, for example, try to make guesses about people based on what they seemed to have in common with other people I knew.
Now someone comes and happens to mention that I “don’t seem to try to understand other people”.
I get upset and defensive because I totally do, this person hasn’t understood me at all!
And in one sense, I’m right—it’s true that there’s a dimension of “trying to understand other people” that I’ve put a lot of effort into, in which I’ve probably invested more than other people have.
And in another sense, the other person is right—while I was good at one dimension of “trying to understand other people”, I was severely underinvested in others. And I had not really even properly acknowledged that “trying to understand other people” had other important dimensions too, because I was justifiably proud of my investment in one of them.
But from the point of view of someone who *had* invested in those other dimensions, they could see the aspects in which I was deficient compared to them, or maybe even compared to the median person. (To some extent I thought that my underinvestment in those other dimensions was *virtuous*, because I was “not making assumptions about people”, which I’d been told was good.) And this underinvestment showed in how I acted.
So the mistake is that if there’s a vaguely defined, multidimensional skill and you are strongly invested in one of its dimensions, you might not realize that you are deficient in the others. And if someone says that you are not good at it, you might understandably get defensive and upset, because you can only think of the evidence which says you’re good at it… while not even realizing the aspects that you’re missing out on, which are obvious to the person who *is* better at them.
Now one could say that the person giving this feedback should be more precise and not make vague, broad statements like “you don’t seem to try to understand other people”. Rather they should make some more specific statement like “you don’t seem to try to make guesses about other people based on how they compare to other people you know”.
And sure, this could be better. But communication is hard; and often the other person *doesn’t* know the exact mistake that you are making. They can’t see exactly what is happening in your mind: they can only see how you behave. And they see you behaving in a way which, to them, looks like you are not trying to understand other people. (And it’s even possible that *they* are deficient in the dimension that *you* are good at, so it doesn’t even occur to them that “trying to understand other people” could mean anything else than what it means to them.)
So they express it in the way that it looks to them, because before you get into a precise discussion about what exactly each of you means by that term, that’s the only way in which they can get their impression across.
It’s natural to get defensive when someone says that you’re bad at something you thought you were good at. But the things we get defensive about, are also things that we frequently have blindspots around. Now if this kind of a thing seems to happen to me again, I try to make an effort to see whether the skill in question might have a dimension that I’ve been neglecting.
Once I’ve calmed down and stopped being defensive, that is.
(see also this very related essay by Ferrett)
The essay “Don’t Fight Your Default Mode Network” is probably the most useful piece of productivity advice that I’ve read in a while.
Basically, “procrastination” during intellectual work is actually often not wasted time, but rather your mind taking the time to process the next step. For example, if I’m writing an essay, I might glance at a different browser tab while I’m in the middle of writing a particular sentence. But often this is actually *not* procrastination; rather it’s my mind stopping to think about the best way to continue that sentence. And this turns out to be a *better* way to work than trying to keep my focus completely on the essay!
Realizing this has changed my attention management from “try to eliminate distractions” to “try to find the kinds of distractions which don’t hijack your train of thought”. If I glance at a browser tab and get sucked into a two-hour argument, then that still damages my workflow. The key is to try to shift your pattern towards distractions like “staring into the distance for a moment”, so that you can take a brief pause without getting pulled into anything different.
I only now made the connection that Sauron lost because he fell prey to the Typical Mind Fallacy (assuming that everyone’s mind works the way your own does). Gandalf in the book version of The Two Towers:
I was thinking of a friend and recalled some pleasant memories with them, and it occurred to me that I have quite a few good memories about them, but I don’t really recall them very systematically. I just sometimes remember them at random. So I thought, what if I wrote down all the pleasant memories of my friend that I could recall?
Not only could I then occasionally re-read that list to get a nice set of pleasant memories, that would also reinforce associations between them, making it more likely that recalling one—or just being reminded of my friend in general—would also bring to mind all the others.
(This was in part inspired by Steve Andreas’s notion of building a self-concept. There you build self-esteem by taking memories of yourself where you exhibited some positive quality, and intentionally associate them together under some heading such as “lovable” or “intelligent”, so that they become interconnected exemplars of a quality that you have rather than being isolated instances.)
So I did, and that usual thing happened where I started out with just three or so particularly salient memories, but then in the process of writing them down my mind generated a few more, until I had quite a long list. It felt really good; now I want to write similar lists about all my close friends.
Interestingly I noticed that the majority of the memories on my list were ones where I’d helped my friend and they’d been happy as a result, rather than the other way around. This does say something about me finding it easier to help people than to ask for help, but might also be related to the finding that I’ve heard quoted, that giving a gift makes people happier than receiving one.
This is a great idea!
I also had somewhat the inclination to do this, when I first read about Anki on Michael Nielsen’s -Aumenting Cognition, he speaks about using Anki to store memories and friends’ characteristics such as food preferences (he talks about this on the section: “The challenges of using Anki to store facts about friends and family”).
I did not do this because I did not want to meddle with Anki and personal stuff but I found another similar solution which is MONICA a “Personal Relationship Manager”, the good thing about it is that it’s open source and easy to set up. I did use it for a bit and found that it was very easy to use and had all the things one may want.
I ended up not going through using the app at the time, but considering the post and the fact that people love when you remember facts about them (I also’d like to remember things about them!) I may pick it up again.
For a few weeks or so, I’ve been feeling somewhat amazed at how much less suffering there seems to be associated with different kinds of pain (emotional, physical, etc.), seemingly as a consequence of doing meditation and related practices. The strength of pain, as measured by something like the intensity of it as an attention signal, seems to be roughly the same as before, but despite being equally strong, it feels much less aversive.
To clarify, this is not during some specific weird meditative state, but feels like a general ongoing adjustment even when I feel otherwise normal (or otherwise like shit).
I can’t help but to wonder whether the difference in intuitions for/against suffering-focused ethics is a consequence of different people’s brains being naturally differently configured with regard to their pain:suffering ratio. That is, some people will experience exactly the same amount of pain, unpleasant emotions etc. during their life as others, but for some people the same intensity of pain will translate to a different intensity of suffering. And then we will have people who say things like “life *is* suffering and possibly a net negative for many if not most” as well as people who say things like “suffering isn’t any big deal and a pretty uninteresting thing to focus on”, staring at each other in mutual incomprehension.
Interesting, I wonder if there is a way to test it, given that it seems hard to measure the pain:suffering ratio of a person directly...
Is there a form of meditation that makes pain more aversive? Then we can have people who say “suffering isn’t any big deal and a pretty uninteresting thing to focus on” do that, and see if they end up agreeing with suffering-focused ethics?
While this is a brilliant idea in the sense of being a novel way to test a hypothesis, trying to reprogram people’s brains so as to make them experience more suffering strikes me as an ethically dubious way of doing the test. :)
I wouldn’t expect just a one-off meditation session where they experienced strong suffering to be enough, but rather I would expect there to be a gradual shift in intuitions after living with an altered ratio for a long enough time.
Regarding measurement of pain:suffering ratio
A possible approach would be to use self-reports (the thing that doctor’s always ask about, pain scale 1-10) vs revealed preferences (how much painkillers were requested? What trade-offs for pain relief do patients choose?).
Obviously this kind of relation is flawed on several levels: Reported pain scale depends a lot on personal experience (very painful events permanently change the scale, ala “I am in so much pain that I cannot walk or concentrate, but compared to my worst experience… let’s say 3?”). Revealed preferences depend a lot on how much people care about the alternatives (e.g. if people have bad health insurance or really important stuff to do they might accept a lot of subjective suffering in order to get out of hospital one day early). Likewise, time preference might enter a lot into revealed preference.
Despite these shortcomings, that’s where I would start thinking about what such a ratio would mean. If one actually did a study with new questionaires, one should definitely ask patients for some examples in order to gauge their personal pain-scale, and combine actual revealed preferences with answers to hypothetical questions “how much money would pain relief be worth to you? How much risk of death? How many days of early hospital release? etc”, even if the offer is not actually on the table.
Apparently there have been a few studies on something like this: “[Long-Term Meditators], compared to novices, had a significant reduction of self-reported unpleasantness, but not intensity, of painful stimuli, while practicing Open Monitoring.”
I dreamt that you could donate LessWrong karma to other LW users. LW was also an airport, and a new user had requested donations because to build a new gate at the airport, your post needed to have at least 60 karma and he had a plan to construct a series of them. Some posts had exactly 60 karma, with titles like “Gate 36 done, let’s move on to the next one—upvote the Gate 37 post!”.
(If you’re wondering what the karma donation mechanism was needed for if users could just upvote the posts normally—I don’t know.)
Apparently the process of constructing gates was separate from connecting them to the security control, and things had stopped at gate 36⁄37 because it needed to be connected up with security first. I got the impression that this was waiting for the security people to get it done.
This paper (Keno Juechems & Christopher Summerfield: Where does value come from? Trends in Cognitive Sciences, 2019) seems interesting from an “understanding human values” perspective.
Some choice quotes:
This framework of having multiple axes representing different goals, and trying to minimize the sum of distances to their setpoints, also reminds me a bit of moridinamael’s Complex Behavior from Simple (Sub)Agents.
Recent papers relevant to earlier posts in my multiagent sequence:
Understanding the Higher-Order Approach to Consciousness. Richard Brown, Hakwan Lau, Joseph E.LeDoux. Trends in Cognitive Sciences, Volume 23, Issue 9, September 2019, Pages 754-768.
Reviews higher-order theories (HOT) of consciousness and their relation to global workspace theories (GWT) of consciousness, suggesting that HOT and GWT are complementary. Consciousness and the Brain, of course, is a GWT theory; whereas HOT theories suggest that some higher-order representation is (also) necessary for us to be conscious of something. I read the HOT models as being closely connected to introspective awareness; e.g. the authors suggest a connection between alexityhmia (unawareness of your emotions) and abnormalities in brain regions related to higher-order representation.
While the HOT theories seem to suggest that you need higher-order representation of something to be conscious of a thing, I would say that you need higher-order representation of something in order to be conscious of having been conscious of something. (Whether being conscious of something without being conscious of being conscious of it can count as being conscious of it, is of course an interesting philosophical question.)
Bridging Motor and Cognitive Control: It’s About Time! Harrison Ritz, Romy Frömer, Amitai Shenhav. Trends in Cognitive Sciences, in press.
I have suggested that control of thought and control of behavior operate on similar principles; this paper argues the same.
From Knowing to Remembering: The Semantic–Episodic Distinction. Louis Renoult, Muireann Irish, Morris Moscovitch, and Michael D. Rugg. Trends in Cognitive Sciences, in press.
In Book summary: Unlocking the Emotional Brain and Building up to an Internal Family Systems model, I referenced models under which a particular event in a person’s life gives rise to a generalized belief schema, and situations which re-activate that belief schema may also partially re-activate recollection of the original event, and vice versa; if something reminds you of a situation you experienced as a child, you may also to some extent reason in the kinds of terms that you did when you were a child and in that situation. This paper discusses connections between episodic memories (e.g., “I remember reading 1984 in Hyde Park yesterday”) and semantic memories (e.g. “1984 was written by George Orwell”), and how activation of one may activate another.
Hypothesis: basically anyone can attract a cult following online, provided that they
1) are a decent writer or speaker
2) are writing/speaking about something which may or may not be particularly original, but does provide at least some value to people who haven’t heard of this kind of stuff before
3) devote a substantial part of their message into confidently talking about how their version of things is the true and correct one, and how everyone who says otherwise is deluded/lying/clueless
There’s a lot of demand for the experience of feeling like you know something unique that sets you apart from all the mundane, unwashed masses.
(This isn’t necessarily a bad thing. As long as the content that’s being peddled is something reasonable, then these people’s followers may get a lot of genuine value from being so enthusiastic about it. Being really enthusiastic almost by definition means that you are going to invest a lot more into internalizing and using the thing, than does someone who goes “meh, that’s old hat” and then never actually does anything with the thing. A lot depends on how sensible the content is—this method probably works equally well with content that’s a net harm to buy into, as it does with content that’s a net good. But of course, the fact that it works basically regardless of what the content is, means that a lot of the content in question will be bad.)
Other common marketing advice that fits into this:
Set up a “bad guy” that you’re against
If you’re in a crowded category, either
Create a new category (e.g. rationality)
Set yourself up as an alternative to number in a category (Pepsi)
Become number one in the category (Jetblue?)
It’s better to provide value that takes away a pain (painkillers) than that adds something that was missing (vitamins)
I’d really like to read more about what you think of this. Another closely related feature they need is:
Content well formatted (The Sequences are a great example of this,The Codex). Of course, blogs are also a good basic idea which allows incremental reading.
Length of the posts? Maybe? I think there may be a case to be made for length helping to generate that cult following since it’s directly related to the amount of time invested by people reading. There are many examples where posts could be summarized by a few paragraphs but instead they go long! (But of course there’s a reason they do so).
Some time back, Julia Wise published the results of a survey asking parents what they had expected parenthood to be like and to what extent their experience matched those expectations. I found those results really interesting and have often referred to them in conversation, and they were also useful to me when I was thinking about whether I wanted to have children myself.
However, that survey was based on only 12 people’s responses, so I thought it would be valuable to get more data. So I’m replicating Julia’s survey, with a few optional quantitative questions added. If you have children, you’re welcome to answer here: https://forms.gle/uETxvX45u3ebDECy5
I’ll publish the results at some point when it looks like there won’t be many more responses.
The link is a link to a facebook webpage telling my that I am about to leave facebook. Is that intentional?
Oh oops, it wasn’t. Fixed, thanks for pointing it out.
So I was doing insight meditation and noticing inconsistencies between my experience and my mental models of what things in my experience meant (stuff like “this feeling means that I’m actively and consciously spending effort… but wait, I don’t really feel like it’s under my control, so that can’t be right”), and feeling like parts of my brain were getting confused as a result...
And then I noticed that if I thought of a cognitive science/psychology-influenced theory of what was going on instead, those confused parts of my mind seemed to grab onto it, and maybe replace their previous models with that one.
Which raised the obvious question of, wait, am I just replacing one set of flawed assumptions with another?
But that would explain the thing which Scott writes about in https://slatestarcodex.com/2018/04/19/gupta-on-enlightenment/ , where e.g. a Muslim who gets enlightened will adopt an Islamic framework to explain it and experience it as a deep truth. Insight meditation involves making the mind confused about what’s going on, and when a mind gets confused, it will grab onto the first coherent explanation it finds.
But if you’re aware of that, and don’t mistake your new set of assumptions for a universal truth, then you can keep investigating your mind and uncovering new inconsistencies in your models, successively tearing each one apart in order to replace them with ever-more accurate ones.
Didn’t expect to see alignment papers to get cited this way in mainstream psychology papers now.
https://www.sciencedirect.com/science/article/abs/pii/S001002772500071X
What could plausibly take us from now to AGI within 10 years?
A friend shared the following question on Facebook:
I replied with some of my thoughts as follows:
Here’s my brief pitch, starting with your point about simulation:
The strength and flexibility of LLMs probably opens up several more routes toward cognitive completeness and what we’d consider impressive creativity.
LLMs can use chain-of-thought sequential processing to do a type of mental simulation. If they are prompted to, or if they “prompt themselves” in a chain of thought system, they can access a rich world model to simulate how different actions are likely to play out. They have to put everything in language, although visual and other modalities can be added either through things like the whiteboard of thought, or by using CoT training directly on those modalities in multimodal foundation models. But language already summarizes a good deal of world models across many modalities, so those improvements may not be necessary.
The primary change that will make LLMs more “creative” in your friends’ sense is letting them think longer and using strategy and training to organize that thinking. There are two cognitive capacities needed to do this. There is no barrier to progress in either direction; they just haven’t received much attention yet.
LLMs don’t have any episodic memory, “snapshot” memory for important experiences. And They’re severely lacking executive functioning, the capacity to keep ourselves on-track and strategically direct our cognition. A human with those impairments would be very little use for complex tasks, let alone doing novel work we’d consider deeply creative.
Both of those things seem actually pretty easy to add. Vector-based databases aren’t quite good enough to be very useful, but they will be improved. One route is a straightforward, computationally-efficient improvement based on human brain function that I won’t mention even though work is probably underway on it somewhere. And there are probably other equally good routes.
The chain-of-thought training applied to o1, r1, Marco o1, and QwQ (and probably soon a whole bunch more) improves organization of chains of thought, adding some amount of executive function. Scaffolding in prompts for things like “where are you in the task? Is this making progress toward the goal? Should we try a different approach?” etc is also possible. This will work better when combined with episodic memory; a human without it couldn’t organize their progress through a complex task—but LLMs now have large context windows that are like better-than-human working memory systems, so better episodic memory might not even be necessary for dramatic improvements.
This is spelled out a little more in Capabilities and alignment of LLM cognitive architectures, although that isn’t as clear or compelling as I’d like. It looks to me like progress is happening apace on that direction.
That’s just one route to “Real AGI” from LLMs/foundation models. There are probably others that are just as easy. Foundation models can now do almost everything humans can in the short term. Making their cognition cumulative like ours seems like more of an unblocking and using their capacities more strategically and effectively, rather than adding any real new cognitive abilities.
Continuous learning, through better episodic memory and/or fine-tuning for facts/skills judged as useful is another low-hanging fruit.
Hoping that we’re more than a decade from transformative AGI now seems wildly optimistic to me. There could be dramatic roadblocks I haven’t foreseen, but most of those would just push it past three years. It could take more than a decade, but banking on that leaves us unprepared for the very short timelines that now seem fairly likely.
While the short timelines are scary, there are also large advantages to this route to AGI, including a relatively slow takeoff and the way that LLMs are almost an oracle AI trained largely to follow instructions. But that’s another story.
That’s a bit more than I meant to write; I’ve been trying to refine an intuitive explanation of why we may be spitting distance from real, transformative AGI, and that served as a useful prompt.
Self-driving cars seem like a useful reference point. Back when cars got unexpectedly good performance at the 2005 and 2007 DARPA grand challenges, there was a lot of hype about how self-driving cars were just around the corner now that they had demonstrated having the basic capability. 17 years later, we’re only at this point (Wikipedia):
And self-driving capability should be vastly easier than general intelligence. Like self-driving, transformative AI also requires reliable worst-case performance rather than just good average-case performance, and there’s usually a surprising amount of detail involved that you need to sort out before you get to that point.
I admit, I’d probably call self-driving cars at this point a solved or nearly-solved problem by Waymo, and the big reason why self-driving cars only now are taking off is basically because of regulatory and liability issues, and I consider a lot of the self-driving car slowdown as evidence that regulation can work to slow down a technology substantially.
(Hmm I was expecting that this would get more upvotes. Too obvious? Not obvious enough?)
It seems to me that o1 and deepseek already do a bunch of the “mental simulation” kind of reasoning, and even previous LLMs did so a good amount if you prompted them to think in chain-of-thoughts, so the core point fell a bit flat for me.
Thanks, that’s helpful. My impression from o1 is that it does something that could be called mental simulation for domains like math where the “simulation” can in fact be represented with just writing (or equations more specifically). But I think that writing is only an efficient format for mental simulation for a very small number of domains.
A morning habit I’ve had for several weeks now is to put some songs on, then spend 5-10 minutes letting the music move my body as it wishes. (Typically this turns into some form of dancing.)
It’s a pretty effective way to get my energy / mood levels up quickly, can recommend.
It’s also easy to effectively timebox it if you’re busy, “I will dance for exactly two songs” serves as its own timer and is often all I have the energy for before I’ve had breakfast. (Today Spotify randomized Nightwish’s Moondance as the third song and boy I did NOT have the blood sugar for that, it sucked me in effectively enough that I did the first 30 seconds but then quickly stopped it after the pace slowed down and it momentarily released its grip on me.)
Janina Fisher’s book “Healing the Fragmented Selves of Trauma Survivors” has an interesting take on Internal Family Systems. She conceptualizes trauma-related parts (subagents) as being primarily associated with the defensive systems of Fight/Flight/Freeze/Submit/Attach.
Here’s how she briefly characterizes the various systems and related behaviors:
Fight: Vigilance. Angry, judgmental, mistrustful, self-destructive, controlling, suicidal, needs to control.
Flight: Escape. Distancer, ambivalent, cannot commit, addictive behavior or being disorganized.
Freeze: Fear. Frozen, terrified, wary, phobic of being seen, agoraphobic, reports panic attacks.
Submit: Shame. Depressed, ashamed, filled with self-hatred, passive, “good girl,” caretaker, self-sacrificing.
Attach: Needy. Desperate, craves rescue & connection, sweet, innocent, wants someone to depend on.
Here’s how she describes a child-like part connected to an “attach” system coming to existence:
Here are how she relates various trauma symptoms to these systems:
And here’s how she describes something that in traditional IFS terms would be described as polarized parts:
I gave this comment a “good facilitation” react but that feels like a slightly noncentral use of it (I associate “good facilitation” more with someone coming in when two other people are already having a conversation). It makes me think that every now and then I’ve seen comments that help clearly distill some central point in a post, in the way that this comment did, and it might be nice to have a separate react for those.
Huh. I woke up feeling like meditation has caused me to no longer have any painful or traumatic memories: or rather all the same memories are still around, but my mind no longer flinches away from them if something happens to make me recall them.
Currently trying to poke around my mind to see whether I could find any memory that would feel strongly aversive, but at most I can find ones that feel a little bit unpleasant.
Obviously can’t yet tell whether some will return to being aversive. But given that this seems to be a result of giving my mind the chance to repeatedly observe that flinching away from things is by itself the thing that makes the things unpleasant, I wouldn’t be too surprised if I’d managed to successfully condition it to stop doing that for the memories. Though I would expect there to be setbacks, the next time that something particularly painful happened or was just generally feeling bad.
This seems similar to my experiences.