Kaj_Sotala

Karma: 49,251

Kaj_Sotala Mar 11, 2025, 6:58 PM
2 points
0
on: Book Review: Affective Neuroscience
Thank you! People keep mentioning Panksepp’s work but I had never gotten around reading it; this was a very nice summary. The described mechanisms felt very plausible and interesting.
primates (including humans) are innately afraid of snakes, spiders,
This has been disputed since then.

Kaj_Sotala Mar 7, 2025, 10:51 AM
20 points
8
on: Kaj’s shortform feed
Something I think about a lot when I see hypotheses based on statistical trends of somewhat obscure variables: I’ve heard it claimed that at one point in Finland, it was really hard to get a disability pension because of depression or other mental health problems, even though it was obvious to many doctors that their patients were too depressed to work. So then some doctors would diagnose those people with back pain instead, since it sounded more like a “real” condition while also being impossible to disprove before ultrasound scans got more common.
I don’t know how big that effect was in practice. But I could imagine a world where it was significant and where someone noticed a trend of back pain diagnoses getting less common while depression diagnoses got more common, and postulating some completely different explanation for the relationship.
More generally, quite a few statistics are probably reporting something different from what they seem to be about. And unless you have deep knowledge about the domain in question, it’ll be impossible to know when that’s the case.

Kaj_Sotala Mar 6, 2025, 2:55 PM
2 points
0
in reply to: Petropolitan’s comment on: Daniel Kokotajlo’s Shortform
Thanks!

Kaj_Sotala Mar 6, 2025, 7:48 AM
9 points
4
in reply to: johnswentworth’s comment on: A Bear Case: My Predictions Regarding AI Progress
I think I’m also around 60-70% for the rough overall picture in the OP being correct.
What links here?
- Surprising LLM reasoning failures make me think we still need qualitative breakthroughs for AGI by Kaj_Sotala (Apr 15, 2025, 3:56 PM; 107 points)

Kaj_Sotala Mar 5, 2025, 12:01 PM
2 points
0
on: On Writing #1
Marc is saying that first you write out your points and conclusion, then you fill in the details. He wants to figure it all out while his mind is buzzing, then justify it later.
Whereas I learn what I think as I write out my ideas in detail. I would say that if you are only jotting down bullet points, you do not yet know what you think.
Might Marc’s mind not work differently from yours?
He could also have done a large part of his thinking in some different way already, e.g. in conversations with people.

Kaj_Sotala Mar 5, 2025, 11:08 AM
6 points
9
in reply to: momom2’s comment on: How to Make Superbabies
There’s also the option that even if this technology is initially funded by the wealthy, learning curves will then drive down its cost as they do for every technology, until it becomes affordable for governments to subsidize its availability for everyone.

Kaj_Sotala Mar 5, 2025, 9:52 AM
11 points
4
in reply to: Jonas Hallgren’s comment on: Statistical Challenges with Making Super IQ babies
In the modern era, the fertility-IQ correlation seems unclear; in some contexts, higher fertility seems to be linked with lower IQ, in other contexts with higher IQ. I have no idea of what it was like in the hunter-gatherer era, but it doesn’t feel like an obviously impossible notion that very high IQs might have had a negative effect on fertility in that time as well.
E.g. because the geniuses tended to get bored with repeatedly doing routine tasks and there wasn’t enough specialization to offload that to others, thus leading to the geniuses having lower status. Plus having an IQ that’s sufficiently higher than that of others can make it hard to relate to them and get along socially, and back then there wouldn’t have been any high-IQ societies like a university or lesswrong.com to find like-minded peers at.

Kaj_Sotala Mar 5, 2025, 7:11 AM
3 points
1
in reply to: mako yass’s comment on: The Milton Friedman Model of Policy Change
I think something doesn’t need to be fundamentally new for the press to turn it into a scary story, e.g. news reports about crime or environmental devastation being on the rise have scared a lot of people quite a bit. You can’t photograph a quantity but you can photograph individuals affected by a thing and make it feel common by repeatedly running stories of different individuals affected.

Kaj_Sotala Mar 4, 2025, 10:48 AM
6 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
I’ve spent enough time staring at LLM chain-of-thoughts now that when I started thinking about a thing for work, I found my thoughts taking the shape of an LLM thinking about how to approach its problem. And that actually felt like a useful systematic way of approaching the problem, so I started writing out that chain of thought like I was an LLM, and that felt valuable in helping me stay focused.
Of course, I had to amuse myself by starting the chain-of-thought with “The user has asked me to...”

Kaj_Sotala Mar 2, 2025, 7:39 AM
2 points
0
in reply to: Mantas Mazeika’s comment on: Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
However, I don’t think this means that their values over hypothetical states of the world is less valuable to study.
Yeah, I don’t mean that this wouldn’t be interesting or valuable to study—sorry for sounding like I did. My meaning was something more in line with Olli’s comment, that this is interesting but that the generalization from the results to “GPT-4o is willing to trade off” etc. sounds too strong to me.

Kaj_Sotala Mar 1, 2025, 12:33 PM
LW: 12 AF: 4
3
AF
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
I don’t know what your views on self-driving cars are, but if you are like me you look at what Waymo is doing and you think “Yep, it’s working decently well now, and they are scaling up fast, seems plausible that in a few years it’ll be working even better and scaled to every major city. The dream of robotaxis will be a reality, at least in the cities of America.”
The example of self-driving cars is actually the biggest one that anchors me to timelines of decades or more. A lot of people’s impression after the 2007 DARPA Grand Challenge seemed to be something like “oh, we seem to know how to solve the problem in principle, now we just need a bit more engineering work to make it reliable and agentic in the real world”. Then actually getting things to be as reliable as required for real agents took a lot longer. So past experience would imply that going from “we know in principle how to make something act intelligently and agentically” to “this is actually a reliable real-world agent” can easily take over a decade.
Another example is that going from the first in-principle demonstration of chain-of-thought to o1 took two years. That’s much shorter than a decade but also a much simpler capability.
For general AI, I would expect the “we know how to solve things in principle” stage to at least be something like “can solve easy puzzles that a normal human can that the AI hasn’t been explicitly trained on”. Whereas with AI, we’re not even there yet. E.g. I tried giving GPT-4.5, DeepSeek R1, o3-mini, and Claude 3.7 with extended thinking a simple sliding square problem, and they all committed an illegal move at one stage or another.
And that’s to say nothing about all the other capabilities that a truly general agent—say one capable of running a startup—would need, like better long-term memory, ability to formulate its own goals and prioritize between them in domains with no objective rules you could follow to guarantee success, etc.. Not only are we lacking convincing in-principle demonstrations of general intelligence within puzzle-like domains, we’re also lacking in-principle demonstrations of these other key abilities.

Kaj_Sotala Mar 1, 2025, 11:06 AM
2 points
0
in reply to: Matrice Jacobine’s comment on: Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
Maybe? We might still have consistent results within this narrow experimental setup, but it’s not clear to me that it would generalize outside that setup.

Kaj_Sotala Mar 1, 2025, 11:00 AM
4 points
2
in reply to: Mantas Mazeika’s comment on: Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
Forced choice settings are commonly used in utility elicitation, e.g. in behavioral economics and related fields.
True. It’s also a standard criticism of those studies that answers to those questions measure what a person would say in response to being asked those questions (or what they’d do within the context of whatever behavioral experiment has been set up), but not necessarily what they’d do in real life when there are many more contextual factors. Likewise, these questions might answer what an LLM with the default persona and system prompt might answer when prompted with only these questions, but don’t necessarily tell us what it’d do when prompted to adopt a different persona, when its context window had been filled with a lot of other information, etc..
The paper does control a bit for framing effects by varying the order of the questions, and notes that different LLMs converge to the same kinds of answers in that kind of neutral default setup, but that doesn’t control for things like “how would 10 000 tokens worth of discussion about this topic with an intellectually sophisticated user affect the answers”, or “how would an LLM value things once a user had given it a system prompt making it act agentically in the pursuit of the user’s goals and it had had some discussions with the user to clarify the interpretation of some of the goals”.
Some models like Claude 3.6 are a bit infamous for very quickly flipping all of their views into agreement with what it thinks the user’s views are, for instance. Results like in the paper could reflect something like “given no other data, the models predict that the median person in their training data would have/prefer views like this” (where ‘training data’ is some combination of the base-model predictions and whatever RLHF etc. has been applied on top of that; it’s a bit tricky to articulate who exactly the “median person” being predicted is, given that they’re reacting to some combination of the person they’re talking with, the people they’ve seen on the Internet, and the behavioral training they’ve gotten).

Kaj_Sotala Mar 1, 2025, 10:53 AM
8 points
3
in reply to: Jiro’s comment on: You can just wear a suit
This reasoning would justify violating any social convention whatsoever. “Refusing to say ‘please’ and ‘thank you’ signals confidence and self-esteem”.
Wrong. I distinguished between conventions that people have a reason to respond negatively to if you violate them (e.g. wearing a suit to the gym or swimming pool which is stupid since it will ruin both your exercise and your suit), and behaviors that just happen to be unusual but not intrinsically negative. Refusing to say ‘please’ and ‘thank you’ would fall squarely in the category that people would have a reason to feel negative about.
My understanding is that fedoras became a sign of cluelessness because they got associated with groups like pick-up artists, which is also an explicit reason to have a negative reaction to them.

Kaj_Sotala Feb 28, 2025, 1:46 PM
14 points
9
in reply to: Jiro’s comment on: You can just wear a suit
Wearing a suit in an inappropriate context is like wearing a fedora. It says “I am socially clueless enough to do random inappropriate things”.
“In an inappropriate context” is ambiguous. It can mean “in a context where people don’t normally wear suits” or it can mean “in a place where people consider it actively wrong to wear a suit”.
There are of course places of the latter type, like it would be very weird to wear a suit in a gym or swimming pool. But I don’t think lsusr would advocate that.
If you just mean “in a context where people don’t normally wear suits”, then wearing a suit in such a context could signal social cluelessness, but it could also signal confidence and self-esteem.

Kaj_Sotala Feb 26, 2025, 10:36 PM
8 points
2
in reply to: deep’s comment on: Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
I don’t have an intuition for whether this is large for a fine-tuning update.
FWIW, OpenAI’s documentation ( https://platform.openai.com/docs/guides/fine-tuning ) says:
To fine-tune a model, you are required to provide at least 10 examples. We typically see clear improvements from fine-tuning on 50 to 100 training examples with gpt-4o-mini and gpt-3.5-turbo, but the right number varies greatly based on the exact use case.
We recommend starting with 50 well-crafted demonstrations and seeing if the model shows signs of improvement after fine-tuning. In some cases that may be sufficient, but even if the model is not yet production quality, clear improvements are a good sign that providing more data will continue to improve the model. No improvement suggests that you may need to rethink how to set up the task for the model or restructure the data before scaling beyond a limited example set.

Kaj_Sotala Feb 25, 2025, 6:44 AM
7 points
0
on: Kaj’s shortform feed
I dreamt that you could donate LessWrong karma to other LW users. LW was also an airport, and a new user had requested donations because to build a new gate at the airport, your post needed to have at least 60 karma and he had a plan to construct a series of them. Some posts had exactly 60 karma, with titles like “Gate 36 done, let’s move on to the next one—upvote the Gate 37 post!”.
(If you’re wondering what the karma donation mechanism was needed for if users could just upvote the posts normally—I don’t know.)
Apparently the process of constructing gates was separate from connecting them to the security control, and things had stopped at gate ³⁶⁄₃₇ because it needed to be connected up with security first. I got the impression that this was waiting for the security people to get it done.

Kaj_Sotala Feb 23, 2025, 8:10 PM
LW: 19 AF: 7
1
AF
on: Have LLMs Generated Novel Insights?
Derya Unutmaz reported that o1-pro came up with a novel idea in the domain of immunotherapy:
Introduction to the Context:
I’m working on developing innovative cancer immunotherapy approaches to address key challenges in the field. Immunotherapy is an exceptionally powerful strategy for curing cancer because it harnesses the body’s immune system—our internal army—and empowers it to recognize and eliminate cancer cells. In this effort, we are focusing on engineering T cells, the immune system’s soldiers and generals, through synthetic biology.
However, significant challenges remain, especially in treating solid tumors like breast cancer. Within the tumor microenvironment, T cells often become exhausted due to the overwhelming number of cancer cells and the suppressive environment created by the tumor. This exhaustion severely limits the effectiveness of these therapies.
To tackle this issue, we employ a cutting-edge model system using 3D bioprinted breast cancer tissue integrated with engineered human T cells. These T cells are reprogrammed through advanced synthetic biology techniques to test and develop solutions for overcoming exhaustion.
Prompt to O1-Pro:
Building on work I’ve previously done and tested with o1-Preview and GPT-4o, I posed the following prompt:
“I’d like you to focus on 3D bioprinted solid tumors as a model to address the T cell exhaustion problem. Specifically, the model should incorporate stroma, as seen in breast cancer, to replicate the tumor microenvironment and explore potential solutions. These solutions could involve technologies like T cell reprogramming, synthetic biology circuits, cytokines, transcription factors related to exhaustion, or metabolic programming. Draw inspiration from other fields, such as Battle Royale games or the immune system’s ability to clear infected cells without triggering autoimmunity. Identify potential pitfalls in developing these therapies and propose alternative approaches. Think outside the box and outline iterative goals that could evolve into full-scale projects. Focus exclusively on in vitro human systems and models.”
Why Battle Royale Games?
You might wonder why I referenced Battle Royale games. That’s precisely the point—I wanted to push the model to think beyond conventional approaches and draw from completely different systems for inspiration. While o1-Preview and GPT-4o were able to generate some interesting ideas based on this concept, but they were mostly what I could also conceive though better most PhD students. In contrast, o1-Pro came up with far more creative and innovative solutions, that left me in awe!
Idea #9: A Remarkable Paradigm
Here, I’m sharing one specific idea, which I’ll call Idea #9 based on its iteration sequence. This idea was exceptional because it proposed an extraordinary paradigm inspired by Battle Royale games but more importantly within the context of deep temporal understanding of biological processes. This was the first time any model explicitly considered the time-dependent nature of biological events—an insight that reflects a remarkably advanced and nuanced understanding!
“Adapt or Fail” Under Escalating Challenges:
Another remarkable aspect of idea #9 was that conceptually it drew from the idea of “adapt or fail” in escalating challenges, directly inspired by Battle Royale mechanics. This was the first time any model could think of it from this perspective. It also emphasized the importance of temporal intervals in reversing or eliminating exhausted T cells. Indeed, this approach mirrors the necessity for T cells to adapt dynamically under pressure and survive progressively tougher challenges, something we would love to model in in vitro systems! One particularly further striking insight was the role of stimulation intervals in preventing exhaustion. Idea #9 suggested that overly short intervals between stimuli might be a key factor driving T cell exhaustion in current therapies. This observation really amazed me with its precision and relevance—because it pinpointed a subtle but critical aspect of T cell activations and development of exhaustion mechanisms.
There’s more behind the link. I have no relevant expertise that would allow me to evaluate how novel this actually was. But immunology is the author’s specialty with his work having close to 30 000 citations on Google Scholar, so I’d assume him to know what he’s talking about.

Kaj_Sotala Feb 23, 2025, 7:43 PM
2 points
0
in reply to: cubefox’s comment on: Viliam’s Shortform
That by itself wouldn’t imply the language model not knowing what the criteria for refusal are, though. It would be simpler to just let the model decide whether it agrees to call the function or not, than to have the function itself implement another check.

Kaj_Sotala Feb 23, 2025, 4:12 PM
18 points
6
on: The Sorry State of AI X-Risk Advocacy, and Thoughts on Doing Better
The history of climate change activism seems like an obvious source to look for ideas. It’s a recent mass movement that has convinced a large part of the general public that something is at least a catastrophic or even an existential risk and put major pressure on governments worldwide. (Even though it probably won’t actually end civilization, many people believe that it will.)
Its failures also seem analogous to what the AI risk movement seems to be facing:
- strong financial incentives to hope that there’s no problem
- action against it having the potential to harm the quality of life of ordinary people (though in the case of novel AI, it’s less about losing things people already have and more about losing the potential for new improvements)
- (sometimes accurate, sometimes not) accusations of fear-mongering and that the concerns are only an excuse for other kinds of social engineering people want to achieve anyway
- overall a degree of collective action that falls significantly short of what the relevant experts believe necessary for stopping the current trajectory of things getting worse