Carl Feynman

Karma: 1,707

I was born in 1962 (so I’m in my 60s). I was raised rationalist, more or less, before we had a name for it. I went to MIT, and have a bachelors degree in philosophy and linguistics, and a masters degree in electrical engineering and computer science. I got married in 1991, and have two kids. I live in the Boston area. I’ve worked as various kinds of engineer: electronics, computer architecture, optics, robotics, software.

Around 1992, I was delighted to discover the Extropians. I’ve enjoyed being in that kind of circles since then. My experience with the Less Wrong community has been “I was just standing here, and a bunch of people gathered, and now I’m in the middle of a crowd.” A very delightful and wonderful crowd, just to be clear.

I‘m signed up for cryonics. I think it has a 5% chance of working, which is either very small or very large, depending on how you think about it.

I may or may not have qualia, depending on your definition. I think that philosophical zombies are possible, and I am one. This is a very unimportant fact about me, but seems to incite a lot of conversation with people who care.

I am reflectively consistent, in the sense that I can examine my behavior and desires, and understand what gives rise to them, and there are no contradictions I‘m aware of. I’ve been that way since about 2015. It took decades of work and I’m not sure if that work was worth it.

Carl Feynman Apr 6, 2025, 12:37 AM
8 points
2
on: Most Questionable Details in ‘AI 2027’
Nitpick: No single organism can destroy the biosphere; at most it can fill its niche & severely disrupt all ecosystems.

Have you read the report on mirror life that came out a few months ago? A mirror bacterium has a niche of “inside any organism that uses carbon-based biochemistry”. At least, it would parasitize all animals, plants, fungi, and the larger Protozoa, and probably kill them. I guess bacteria and viruses would be left. I bet that a reasonably smart superintelligence could figure out a way to get them too.

Carl Feynman Mar 22, 2025, 10:47 PM
4 points
0
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
Quite right. AI safety is moving very quickly and doesn’t have any methods that are well-understood enough to merit a survey article. Those are for things that have a large but scattered literature, with maybe a couple of dozen to a hundred papers that need surveying. That takes a few years to accumulate.

Carl Feynman Mar 22, 2025, 4:00 PM
2 points
0
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
Could you give an example of the sort of distinction you’re pointing at? Because I come to completely the opposite conclusion.

Part of my job is applied mathematics. I’d rather read a paper applying one technique to a variety of problems, than a paper applying a variety of techniques to one problem. Seeing the technique used on several problems lets me understand how and when to apply it. Seeing several techniques on the same problem tells me the best way to solve that particular problem, but I’ll probably never run into that particular problem in my work.
But that’s just me; presumably you want something else out of reading the literature. I would be interested to know what exactly.

Carl Feynman Mar 10, 2025, 4:30 PM
7 points
1
in reply to: ChristianKl’s comment on: So how well is Claude playing Pokémon?
When I say Pokémon-type games, I don’t mean games recounting the adventures of Ash Ketchum and Pikachu. I mean games with a series of obstacles set in a large semi-open world, with things you can carry, a small set of available actions at each point, and a goal of progressing past the obstacles. Such games can be manufactured in unlimited quantities by a program. They can also be “peopled” by simple LLMs, for increased complexity. They don’t actually have to be fun to play or look at, so the design requirements are loose.

There have been attempts at reinforcement learning using unlimited computer-generated games. They haven’t worked that well. I think the key feature that favors Pokémon-like games is that when the player dies or gets stuck, they can go back to the beginning and try again. This rewards trial-and-error learning to get past obstacles, keeping a long-term memory, and to re-plan your approach when something doesn’t work. These are capabilities in which current LLMs are notably lacking.
Another way of saying what Claude’s missing skill is: managing long-term memory. You need to remember important stuff, forget minor stuff, summarize things, and realize when a conclusion in your memory is wrong and needs correction.

Carl Feynman Mar 10, 2025, 12:48 AM
2 points
0
in reply to: ChristianKl’s comment on: So how well is Claude playing Pokémon?
True. I was generalizing it to a system that tries to solve lots of Pokémon-like tasks in various artificial worlds, rather than just expecting it to solve Pokémon over and over. But I didn’t say that, I just imagined in my mind and assumed everyone else would too. Thank you for making it explicit!

Carl Feynman Mar 9, 2025, 6:05 PM
6 points
0
on: A model of the final phase: the current frontier AIs as de facto CEOs of their own companies
This is an important case to think about. I think it is understudied. What separates current AIs from the CEO role? And how long will it take? I see three things:
- Long term thinking, agency, the ability to remember things, not going crazy in an hour or two. It seems to me like this is all the same problem, in the sense that I think one innovation will solve all of them. This has a lot of effort focused on it. I feel like it’s been a known big problem since GPT-4 and Sydney/Bing, 2 ¹⁄₂ years ago. So, by the Lindy principle, it should be another 2 ¹⁄₂ years until it is solved.
- Persuasiveness. I’ve known a few CEOs in my life; they were all more persuasive than average, and one was genuinely unearthly in her ability to convince people of things. LLMs have been steadily increasing in persuasiveness, and are now par-human. So I think scaling will take care of this. Perhaps a year or two?
- Experience. I don’t know how much of this can be inculcated with the training data, and how much of it requires actually managing people and products and thinking about what can go wrong. Every CEO has to deal with individual subordinates, customers, and counterparties. How much of the job is learning the ins and outs of those particular people? Or does it suffice to have spent a thousand subjective years reading about a million people? So we may already have this.
(Side point: As an engineer watching CEOs, I am amazed by their ability to take a few scattered hints spread across their very busy days, and assemble them into a theory of what’s going on and what to do about it. They’re willing to take what I would consider intolerably flimsy evidence and act on it. When this doesn’t work, its called “jumping to conclusions”, when it does work it’s called “a genius move”. AIs should be good at this if you turn up their temperature.)

Carl Feynman Mar 9, 2025, 5:15 PM
14 points
2
on: Phoenix Rising
Eyes are exactly the worst tissue in which to express luciferase. Lighting the eyeball from within prevents seeing things outside the eye, causing blindness. Make your cat glow anywhere else!
Epistemic status: I know you’re kidding. So am I.

Carl Feynman Mar 9, 2025, 5:01 PM
6 points
−3
in reply to: SpectrumDT’s comment on: Why it’s so hard to talk about Consciousness
My charity extends no further than the human race. Once in a while I think about animal ethics and decide that no, I still don’t care enough to make an effort.
A basic commitment of my charity group from the beginning: no money that benefits things other than people. We don’t donate to benefit political groups, organizations, arts, animals, or the natural world. I’m good with that. Members of the group may of course donate elsewhere, and generally do.
We’ve been doing this since 1998, decades before Effective Altruism was a thing. I don’t have a commitment to Effective Altruism the movement, just to altruism which is effective.

Carl Feynman Mar 9, 2025, 4:24 PM
4 points
1
in reply to: Nate Showell’s comment on: What is the best / most proper definition of “Feeling the AGI” there is?
It seems to me that I have done a lot of careful thinking about timelines, and that I also feel the AGI. Why can’t you have a careful understanding what timelines we should expect, and also have an emotional reaction to that? Reasonably coming to the conclusion that many things will change greatly in the next few years deserves a reaction.

Carl Feynman Mar 9, 2025, 2:49 PM
3 points
0
on: So how well is Claude playing Pokémon?
A task like this, at which the AI is lousy but not hopeless, is an excellent feedback signal for RL. It’s also an excellent feedback signal for “grad student descent”: have a human add mechanisms, and see if Claude gets better. This is a very good sign for capabilities, unfortunately.

Carl Feynman Mar 5, 2025, 1:37 PM
5 points
0
in reply to: SpectrumDT’s comment on: Why it’s so hard to talk about Consciousness
You would suppose wrong! My wife and I belong to a group of a couple of dozen people that investigates charities, picks the best ones, and sends them big checks. I used to participate more, but now I outsource all the effort to my wife. I wasn’t contributing much to the choosing process. I just earn the money 🙂.
What does this have to do with my camp#1 intuitions?

Carl Feynman Mar 4, 2025, 5:18 PM
4 points
−1
in reply to: SpectrumDT’s comment on: Why it’s so hard to talk about Consciousness
I don’t think enjoyment and suffering are arbitrary or unimportant. But I do think they’re nebulous. They don’t have crisp, definable, generally agreed-upon properties. We have to deal with them anyway.
I don’t reason about animal ethics; I just follow standard American cultural standards. And I think ethics matters because it helps us live a virtuous and rewarding life.

Is that helpful?

Carl Feynman Mar 4, 2025, 2:51 AM
4 points
0
in reply to: dirk’s comment on: Why it’s so hard to talk about Consciousness
Well, I’m glad you’ve settled the nature of qualia. There’s a discussion downthread, between TAG and Signer, which contains several thousand words of philosophical discussion of qualia. What a pity they didn’t think to look in Wikipedia, which settles the question!
Seriously, I definitely have sensations. I just think some people experience an extra thing on top of sensations, which they think is an indissoluble part of cognition, and which causes them to find some things intuitive that I find incomprehensible.

Carl Feynman Mar 4, 2025, 2:31 AM
3 points
0
in reply to: espoire’s comment on: Why it’s so hard to talk about Consciousness
Well, I can experience happiness and sadness, and all the usual other emotions, so I just always assumed that was enough to make me a morally significant entity.

Carl Feynman Mar 4, 2025, 2:12 AM
3 points
0
in reply to: SpectrumDT’s comment on: Why it’s so hard to talk about Consciousness
With the lawn mower robot, we are able to say what portions of its construction and software are responsible for its charging-station-seeking behavior, in a well-understood mechanistic way. Presumably, if we knew more about the construction of the human mind, we’d be able to describe the mechanisms responsible for human enjoyment of eating and resting. Are the two mechanisms similar enough that it makes sense to refer to the robot enjoying things? I think that the answer is (a) we don’t know, (b) probably not, and (c) there is no fact of the matter that will allow us to settle the question. It’s just a question of whether two things, more or less similar, should be lumped into the same category. On the other hand, thinking of my dogs as enjoying eating and resting is pretty clearly sensible, because it seems to predict their behavior well, and because the mechanisms involved are probably quite similar in the two species.

Carl Feynman Mar 1, 2025, 3:35 PM
4 points
0
in reply to: james oofou’s comment on: james oofou’s Shortform
An interesting and important question.

We have data about how problem-solving ability scales with reasoning time for a fixed model. This isn’t your question, but it’s related. It’s pretty much logarithmic, IIRC.

The important question is, how far can we push the technique whereby reasoning models are trained? They are trained by having them solve a problem with chains of thought (CoT), and then having them look at their own CoT, and ask “how could I have thought that faster?” It’s unclear how far this technique can be pushed (at least to those of us outside the main AI labs).
The known scaling principles are unsatisfying from the point of view of someone who actually wants to know what will happen next. They can predict numbers like score on a certain test, or residual perplexity. But they can’t predict the emergence of new abilities like “can translate languages” or “can tell a joke” or “can take over the world.”

By the way, I wouldn’t put too much weight on any claims about “marginal cost of training DeepSeek R1 over DeepSeek v3”. DeepSeek has a track record of understating how much effort it took to do something. I’m not saying it’s actual dishonesty (although it might be) but it’s at least not counting costs that other companies include, so their estimates come out apparently much lower than other people.

Carl Feynman Feb 28, 2025, 1:18 AM
13 points
3
in reply to: LWLW’s comment on: LWLW’s Shortform
For me, depression has been independent of the probability of doom. I’ve definitely been depressed, but I’ve been pretty cheerful for the past few years, even as the apparent probability of near-term doom has been mounting steadily. I did stop working on AI, and tried to talk my friends out of it, which was about all I could do. I decided not to worry about things I can’t affect, which has clarified my mind immensely.
The near-term future does indeed look very bright.

Carl Feynman Feb 28, 2025, 1:04 AM
3 points
0
in reply to: LWLW’s comment on: LWLW’s Shortform
I am in violent agreement. Nowhere did I say that MuZero could learn a world model as complicated as those LLMs currently enjoy. But it could learn continuously, and execute pretty complex strategies. I don’t know how to combine that with the breadth of knowledge or cleverness of LLMs, but if we could, we’d be in trouble.

Carl Feynman Feb 27, 2025, 7:05 PM
2 points
0
in reply to: Carl Feynman’s comment on: LWLW’s Shortform
Whoops, meant MuZero instead of AlphaZero.

Carl Feynman Feb 27, 2025, 7:01 PM
11 points
0
in reply to: LWLW’s comment on: LWLW’s Shortform
You shouldn’t worry about whether something “is AGI”; it’s an I’ll-defined concept. I agree that current models are lacking the ability to accomplish long-term tasks in the real world, and this keeps them safe. But I don’t think this is permanent, for two reasons.
Current large-language-model type AI is not capable of continuous learning, it is true. But AIs which are capable of it have been built. AlphaZero is perhaps the best example; it learns to play games to a superhuman level in a few hours. It’s a topic of current research to try to combine them.
Moreover, tool-type AIs tend to be developed to provide agency, because it’s more useful to direct an agent than it is a tool. This is a more fully fleshed out here: https://gwern.net/tool-ai
Much of my probability of non-doom is resting on people somehow not developing agents.