Carl Feynman

Karma: 1,693

I was born in 1962 (so I’m in my 60s). I was raised rationalist, more or less, before we had a name for it. I went to MIT, and have a bachelors degree in philosophy and linguistics, and a masters degree in electrical engineering and computer science. I got married in 1991, and have two kids. I live in the Boston area. I’ve worked as various kinds of engineer: electronics, computer architecture, optics, robotics, software.

Around 1992, I was delighted to discover the Extropians. I’ve enjoyed being in that kind of circles since then. My experience with the Less Wrong community has been “I was just standing here, and a bunch of people gathered, and now I’m in the middle of a crowd.” A very delightful and wonderful crowd, just to be clear.

I‘m signed up for cryonics. I think it has a 5% chance of working, which is either very small or very large, depending on how you think about it.

I may or may not have qualia, depending on your definition. I think that philosophical zombies are possible, and I am one. This is a very unimportant fact about me, but seems to incite a lot of conversation with people who care.

I am reflectively consistent, in the sense that I can examine my behavior and desires, and understand what gives rise to them, and there are no contradictions I‘m aware of. I’ve been that way since about 2015. It took decades of work and I’m not sure if that work was worth it.

Carl Feynman Mar 10, 2025, 4:30 PM
7 points
1
in reply to: ChristianKl’s comment on: So how well is Claude playing Pokémon?
When I say Pokémon-type games, I don’t mean games recounting the adventures of Ash Ketchum and Pikachu. I mean games with a series of obstacles set in a large semi-open world, with things you can carry, a small set of available actions at each point, and a goal of progressing past the obstacles. Such games can be manufactured in unlimited quantities by a program. They can also be “peopled” by simple LLMs, for increased complexity. They don’t actually have to be fun to play or look at, so the design requirements are loose.

There have been attempts at reinforcement learning using unlimited computer-generated games. They haven’t worked that well. I think the key feature that favors Pokémon-like games is that when the player dies or gets stuck, they can go back to the beginning and try again. This rewards trial-and-error learning to get past obstacles, keeping a long-term memory, and to re-plan your approach when something doesn’t work. These are capabilities in which current LLMs are notably lacking.
Another way of saying what Claude’s missing skill is: managing long-term memory. You need to remember important stuff, forget minor stuff, summarize things, and realize when a conclusion in your memory is wrong and needs correction.

Carl Feynman Mar 10, 2025, 12:48 AM
2 points
0
in reply to: ChristianKl’s comment on: So how well is Claude playing Pokémon?
True. I was generalizing it to a system that tries to solve lots of Pokémon-like tasks in various artificial worlds, rather than just expecting it to solve Pokémon over and over. But I didn’t say that, I just imagined in my mind and assumed everyone else would too. Thank you for making it explicit!

Carl Feynman Mar 9, 2025, 6:05 PM
6 points
0
on: A model of the final phase: the current frontier AIs as de facto CEOs of their own companies
This is an important case to think about. I think it is understudied. What separates current AIs from the CEO role? And how long will it take? I see three things:
- Long term thinking, agency, the ability to remember things, not going crazy in an hour or two. It seems to me like this is all the same problem, in the sense that I think one innovation will solve all of them. This has a lot of effort focused on it. I feel like it’s been a known big problem since GPT-4 and Sydney/Bing, 2 ¹⁄₂ years ago. So, by the Lindy principle, it should be another 2 ¹⁄₂ years until it is solved.
- Persuasiveness. I’ve known a few CEOs in my life; they were all more persuasive than average, and one was genuinely unearthly in her ability to convince people of things. LLMs have been steadily increasing in persuasiveness, and are now par-human. So I think scaling will take care of this. Perhaps a year or two?
- Experience. I don’t know how much of this can be inculcated with the training data, and how much of it requires actually managing people and products and thinking about what can go wrong. Every CEO has to deal with individual subordinates, customers, and counterparties. How much of the job is learning the ins and outs of those particular people? Or does it suffice to have spent a thousand subjective years reading about a million people? So we may already have this.
(Side point: As an engineer watching CEOs, I am amazed by their ability to take a few scattered hints spread across their very busy days, and assemble them into a theory of what’s going on and what to do about it. They’re willing to take what I would consider intolerably flimsy evidence and act on it. When this doesn’t work, its called “jumping to conclusions”, when it does work it’s called “a genius move”. AIs should be good at this if you turn up their temperature.)

Carl Feynman Mar 9, 2025, 5:15 PM
14 points
2
on: Phoenix Rising
Eyes are exactly the worst tissue in which to express luciferase. Lighting the eyeball from within prevents seeing things outside the eye, causing blindness. Make your cat glow anywhere else!
Epistemic status: I know you’re kidding. So am I.

Carl Feynman Mar 9, 2025, 5:01 PM
6 points
−3
in reply to: SpectrumDT’s comment on: Why it’s so hard to talk about Consciousness
My charity extends no further than the human race. Once in a while I think about animal ethics and decide that no, I still don’t care enough to make an effort.
A basic commitment of my charity group from the beginning: no money that benefits things other than people. We don’t donate to benefit political groups, organizations, arts, animals, or the natural world. I’m good with that. Members of the group may of course donate elsewhere, and generally do.
We’ve been doing this since 1998, decades before Effective Altruism was a thing. I don’t have a commitment to Effective Altruism the movement, just to altruism which is effective.

Carl Feynman Mar 9, 2025, 4:24 PM
4 points
1
in reply to: Nate Showell’s comment on: What is the best / most proper definition of “Feeling the AGI” there is?
It seems to me that I have done a lot of careful thinking about timelines, and that I also feel the AGI. Why can’t you have a careful understanding what timelines we should expect, and also have an emotional reaction to that? Reasonably coming to the conclusion that many things will change greatly in the next few years deserves a reaction.

Carl Feynman Mar 9, 2025, 2:49 PM
2 points
0
on: So how well is Claude playing Pokémon?
A task like this, at which the AI is lousy but not hopeless, is an excellent feedback signal for RL. It’s also an excellent feedback signal for “grad student descent”: have a human add mechanisms, and see if Claude gets better. This is a very good sign for capabilities, unfortunately.

Carl Feynman Mar 5, 2025, 1:37 PM
5 points
0
in reply to: SpectrumDT’s comment on: Why it’s so hard to talk about Consciousness
You would suppose wrong! My wife and I belong to a group of a couple of dozen people that investigates charities, picks the best ones, and sends them big checks. I used to participate more, but now I outsource all the effort to my wife. I wasn’t contributing much to the choosing process. I just earn the money 🙂.
What does this have to do with my camp#1 intuitions?

Carl Feynman Mar 4, 2025, 5:18 PM
4 points
−1
in reply to: SpectrumDT’s comment on: Why it’s so hard to talk about Consciousness
I don’t think enjoyment and suffering are arbitrary or unimportant. But I do think they’re nebulous. They don’t have crisp, definable, generally agreed-upon properties. We have to deal with them anyway.
I don’t reason about animal ethics; I just follow standard American cultural standards. And I think ethics matters because it helps us live a virtuous and rewarding life.

Is that helpful?

Carl Feynman Mar 4, 2025, 2:51 AM
4 points
0
in reply to: dirk’s comment on: Why it’s so hard to talk about Consciousness
Well, I’m glad you’ve settled the nature of qualia. There’s a discussion downthread, between TAG and Signer, which contains several thousand words of philosophical discussion of qualia. What a pity they didn’t think to look in Wikipedia, which settles the question!
Seriously, I definitely have sensations. I just think some people experience an extra thing on top of sensations, which they think is an indissoluble part of cognition, and which causes them to find some things intuitive that I find incomprehensible.

Carl Feynman Mar 4, 2025, 2:31 AM
3 points
0
in reply to: espoire’s comment on: Why it’s so hard to talk about Consciousness
Well, I can experience happiness and sadness, and all the usual other emotions, so I just always assumed that was enough to make me a morally significant entity.

Carl Feynman Mar 4, 2025, 2:12 AM
3 points
0
in reply to: SpectrumDT’s comment on: Why it’s so hard to talk about Consciousness
With the lawn mower robot, we are able to say what portions of its construction and software are responsible for its charging-station-seeking behavior, in a well-understood mechanistic way. Presumably, if we knew more about the construction of the human mind, we’d be able to describe the mechanisms responsible for human enjoyment of eating and resting. Are the two mechanisms similar enough that it makes sense to refer to the robot enjoying things? I think that the answer is (a) we don’t know, (b) probably not, and (c) there is no fact of the matter that will allow us to settle the question. It’s just a question of whether two things, more or less similar, should be lumped into the same category. On the other hand, thinking of my dogs as enjoying eating and resting is pretty clearly sensible, because it seems to predict their behavior well, and because the mechanisms involved are probably quite similar in the two species.

Carl Feynman Mar 1, 2025, 3:35 PM
4 points
0
in reply to: james oofou’s comment on: james oofou’s Shortform
An interesting and important question.

We have data about how problem-solving ability scales with reasoning time for a fixed model. This isn’t your question, but it’s related. It’s pretty much logarithmic, IIRC.

The important question is, how far can we push the technique whereby reasoning models are trained? They are trained by having them solve a problem with chains of thought (CoT), and then having them look at their own CoT, and ask “how could I have thought that faster?” It’s unclear how far this technique can be pushed (at least to those of us outside the main AI labs).
The known scaling principles are unsatisfying from the point of view of someone who actually wants to know what will happen next. They can predict numbers like score on a certain test, or residual perplexity. But they can’t predict the emergence of new abilities like “can translate languages” or “can tell a joke” or “can take over the world.”

By the way, I wouldn’t put too much weight on any claims about “marginal cost of training DeepSeek R1 over DeepSeek v3”. DeepSeek has a track record of understating how much effort it took to do something. I’m not saying it’s actual dishonesty (although it might be) but it’s at least not counting costs that other companies include, so their estimates come out apparently much lower than other people.

Carl Feynman Feb 28, 2025, 1:18 AM
13 points
3
in reply to: LWLW’s comment on: LWLW’s Shortform
For me, depression has been independent of the probability of doom. I’ve definitely been depressed, but I’ve been pretty cheerful for the past few years, even as the apparent probability of near-term doom has been mounting steadily. I did stop working on AI, and tried to talk my friends out of it, which was about all I could do. I decided not to worry about things I can’t affect, which has clarified my mind immensely.
The near-term future does indeed look very bright.

Carl Feynman Feb 28, 2025, 1:04 AM
3 points
0
in reply to: LWLW’s comment on: LWLW’s Shortform
I am in violent agreement. Nowhere did I say that MuZero could learn a world model as complicated as those LLMs currently enjoy. But it could learn continuously, and execute pretty complex strategies. I don’t know how to combine that with the breadth of knowledge or cleverness of LLMs, but if we could, we’d be in trouble.

Carl Feynman Feb 27, 2025, 7:05 PM
2 points
0
in reply to: Carl Feynman’s comment on: LWLW’s Shortform
Whoops, meant MuZero instead of AlphaZero.

Carl Feynman Feb 27, 2025, 7:01 PM
11 points
0
in reply to: LWLW’s comment on: LWLW’s Shortform
You shouldn’t worry about whether something “is AGI”; it’s an I’ll-defined concept. I agree that current models are lacking the ability to accomplish long-term tasks in the real world, and this keeps them safe. But I don’t think this is permanent, for two reasons.
Current large-language-model type AI is not capable of continuous learning, it is true. But AIs which are capable of it have been built. AlphaZero is perhaps the best example; it learns to play games to a superhuman level in a few hours. It’s a topic of current research to try to combine them.
Moreover, tool-type AIs tend to be developed to provide agency, because it’s more useful to direct an agent than it is a tool. This is a more fully fleshed out here: https://gwern.net/tool-ai
Much of my probability of non-doom is resting on people somehow not developing agents.

Carl Feynman Feb 27, 2025, 12:25 AM
6 points
0
in reply to: LWLW’s comment on: LWLW’s Shortform
Welcome to Less Wrong. Sometimes I like to go around engaging with new people, so that’s what I’m doing.
On a sentence-by-sentence basis, your post is generally correct. It seems like you’re disagreeing with something you’ve read or heard. But I don’t know what you read, so I can’t understand what you’re arguing for or against. I could guess, but it would be better if you just said.

Carl Feynman Feb 19, 2025, 10:06 PM
10 points
1
on: Programming Language Early Funding?
I work for a company that developed its own programming language and has been selling it for over twenty years for a great deal of money. For many of those twenty years, I worked in the group developing the language. Before working for my current employer, I participated in several language development efforts. I say this not in order to toot my own horn, but to indicate that what I say has some weight of experience behind it.
There is no way to get the funding you want. I am sorry to tell you this.
From a funder’s point of view, there are several effects that make this a bad bet. First, the difference between reasonably up-to-date general-purpose languages does not seem to be that large. This is as measured by actual effectiveness of projects (manager view) as opposed to whether a language is fun to use (engineer view). So presumably a new language will not offer a large improvement. If your language meets a combination of needs not served by any existing language, that may be a way around this. Second, every person who invents a new programming language thinks it will be a marvelous advance over all existing languages. It’s like how every parent thinks their baby is the cutest. Some of them are actually right, but since all of them think this, a funder has no way of telling which is which. Third, as you note above, it is hard to monetize a new language. This means that the capitalist startup system will not be helpful to you. Fourth, much of language design is a balancing of imponderables (simplicity vs power, garbage collection vs control, etc.) There’s no way to measure these before funding the project, so no way to make confident predictions of whether a new language will be an advance. So the backing of a project is even more of a leap of faith than it usually is. (Some of this risk can be retired by releasing an interpreter and documentation before writing a compiler.)
It’s lousy and it sucks. Sorry.

Carl Feynman Jan 20, 2025, 11:44 PM
4 points
3
in reply to: dirk’s comment on: Why it’s so hard to talk about Consciousness
Well, let me quote Wikipedia:
Much of the debate over the importance of qualia hinges on the definition of the term, and various philosophers emphasize or deny the existence of certain features of qualia. Some philosophers of mind, like Daniel Dennett, argue that qualia do not exist. Other philosophers, as well as neuroscientists and neurologists, believe qualia exist and that the desire by some philosophers to disregard qualia is based on an erroneous interpretation of what constitutes science.
If it was that easy to understand, we wouldn’t be here arguing about it. My claim is that arguments about qualia are (partially) caused by people actually having different cognitive mechanisms that produce different intuitions about how experience works.