Carl Feynman

Karma: 1,725

I was born in 1962 (so I’m in my 60s). I was raised rationalist, more or less, before we had a name for it. I went to MIT, and have a bachelors degree in philosophy and linguistics, and a masters degree in electrical engineering and computer science. I got married in 1991, and have two kids. I live in the Boston area. I’ve worked as various kinds of engineer: electronics, computer architecture, optics, robotics, software.

Around 1992, I was delighted to discover the Extropians. I’ve enjoyed being in that kind of circles since then. My experience with the Less Wrong community has been “I was just standing here, and a bunch of people gathered, and now I’m in the middle of a crowd.” A very delightful and wonderful crowd, just to be clear.

I‘m signed up for cryonics. I think it has a 5% chance of working, which is either very small or very large, depending on how you think about it.

I may or may not have qualia, depending on your definition. I think that philosophical zombies are possible, and I am one. This is a very unimportant fact about me, but seems to incite a lot of conversation with people who care.

I am reflectively consistent, in the sense that I can examine my behavior and desires, and understand what gives rise to them, and there are no contradictions I‘m aware of. I’ve been that way since about 2015. It took decades of work and I’m not sure if that work was worth it.

Carl Feynman Feb 27, 2025, 7:01 PM
11 points
0
in reply to: LWLW’s comment on: LWLW’s Shortform
You shouldn’t worry about whether something “is AGI”; it’s an I’ll-defined concept. I agree that current models are lacking the ability to accomplish long-term tasks in the real world, and this keeps them safe. But I don’t think this is permanent, for two reasons.
Current large-language-model type AI is not capable of continuous learning, it is true. But AIs which are capable of it have been built. AlphaZero is perhaps the best example; it learns to play games to a superhuman level in a few hours. It’s a topic of current research to try to combine them.
Moreover, tool-type AIs tend to be developed to provide agency, because it’s more useful to direct an agent than it is a tool. This is a more fully fleshed out here: https://gwern.net/tool-ai
Much of my probability of non-doom is resting on people somehow not developing agents.

Carl Feynman Feb 27, 2025, 12:25 AM
6 points
0
in reply to: LWLW’s comment on: LWLW’s Shortform
Welcome to Less Wrong. Sometimes I like to go around engaging with new people, so that’s what I’m doing.
On a sentence-by-sentence basis, your post is generally correct. It seems like you’re disagreeing with something you’ve read or heard. But I don’t know what you read, so I can’t understand what you’re arguing for or against. I could guess, but it would be better if you just said.

Carl Feynman Feb 19, 2025, 10:06 PM
11 points
1
on: Programming Language Early Funding?
I work for a company that developed its own programming language and has been selling it for over twenty years for a great deal of money. For many of those twenty years, I worked in the group developing the language. Before working for my current employer, I participated in several language development efforts. I say this not in order to toot my own horn, but to indicate that what I say has some weight of experience behind it.
There is no way to get the funding you want. I am sorry to tell you this.
From a funder’s point of view, there are several effects that make this a bad bet. First, the difference between reasonably up-to-date general-purpose languages does not seem to be that large. This is as measured by actual effectiveness of projects (manager view) as opposed to whether a language is fun to use (engineer view). So presumably a new language will not offer a large improvement. If your language meets a combination of needs not served by any existing language, that may be a way around this. Second, every person who invents a new programming language thinks it will be a marvelous advance over all existing languages. It’s like how every parent thinks their baby is the cutest. Some of them are actually right, but since all of them think this, a funder has no way of telling which is which. Third, as you note above, it is hard to monetize a new language. This means that the capitalist startup system will not be helpful to you. Fourth, much of language design is a balancing of imponderables (simplicity vs power, garbage collection vs control, etc.) There’s no way to measure these before funding the project, so no way to make confident predictions of whether a new language will be an advance. So the backing of a project is even more of a leap of faith than it usually is. (Some of this risk can be retired by releasing an interpreter and documentation before writing a compiler.)
It’s lousy and it sucks. Sorry.

Carl Feynman Jan 20, 2025, 11:44 PM
4 points
3
in reply to: dirk’s comment on: Why it’s so hard to talk about Consciousness
Well, let me quote Wikipedia:
Much of the debate over the importance of qualia hinges on the definition of the term, and various philosophers emphasize or deny the existence of certain features of qualia. Some philosophers of mind, like Daniel Dennett, argue that qualia do not exist. Other philosophers, as well as neuroscientists and neurologists, believe qualia exist and that the desire by some philosophers to disregard qualia is based on an erroneous interpretation of what constitutes science.
If it was that easy to understand, we wouldn’t be here arguing about it. My claim is that arguments about qualia are (partially) caused by people actually having different cognitive mechanisms that produce different intuitions about how experience works.

Carl Feynman Jan 3, 2025, 6:51 PM
4 points
3
in reply to: Terence Coelho’s comment on: notfnofn’s Shortform
Humans continue to get very offended if they find out they are talking to an AI
In my limited experience of phone contact with AIs, this is only true for distinctly subhuman AIs. Then I emotionally react like I am talking to someone who is being deliberately obtuse, and become enraged. I’m not entirely clear on why I have this emotional reaction, but it’s very strong. Perhaps it is related to the Uncanny Valley effect. On the other hand, I’ve dealt with phone AIs that (acted like they) understood me, and we’ve concluded a pleasant and businesslike interaction. I may be typical-minding here, but I suspect that most people will only take offense if they run into the first kind of AI.
Perhaps this is related: I felt a visceral uneasiness dealing with chat-mode LLMs, until I tried Claude, which I found agreeable and helpful. Now I have a claude.ai subscription. Once again, I don’t understand the emotional difference.
I’m 62 years old, which may have something to do with it. I can feel myself being less mentally flexible than I was decades ago, and I notice myself slipping into crotchety-old-man mode more often. It’s a problem that requires deliberate effort to overcome.

Carl Feynman Jan 3, 2025, 3:49 PM
2 points
0
on: The subset parity learning problem: much more than you wanted to know
Has anybody tried actual humans or smart LLMs? It would be interesting to know what methods people actually use.

Carl Feynman Jan 3, 2025, 3:25 PM
2 points
0
on: A pragmatic story about where we get our priors
If we’re being pragmatic, the priors we had at birth almost don’t matter. A few observations will overwhelm any reasonable prior. As long as we don’t assign zero probability to anything that can actually happen, the shape of the prior makes no practical difference.

Carl Feynman Dec 31, 2024, 9:41 PM
12 points
2
on: My AGI safety research—2024 review, ’25 plans
including probably reworking some of my blog post ideas into a peer-reviewed paper for a neuroscience journal this spring.
I think this is a great idea. It will broadcast your ideas to an audience prepared to receive them. You can leave out the “friendly AI” motivation and your ideas will stand on their own as a theory of (some of) cognition.

Carl Feynman Dec 31, 2024, 8:44 PM
5 points
0
on: Could orcas be (trained to be) smarter than humans? 
Do we have a sense for how much of the orca brain is specialized for sonar? About a third of the human brain is specialized to visual perception. If sonar is harder than vision, evolution might have dedicated more of the orca brain to it. On the other hand, orcas don’t need a bunch of brain for manual dexterity, like us.
In humans, the prefrontal cortex is dedicated to “higher” forms of thinking. But evolution slides functions around on the cortical surface, and (Claude tells me) association areas like the prefrontal cortex are particularly prone to this. Just looking for the volume of the prefrontal cortex won’t tell you how much actual thought goes on there.

Carl Feynman Dec 27, 2024, 8:03 PM
4 points
3
on: Review: Planecrash
All the pictures are missing for me.

Carl Feynman Dec 26, 2024, 7:45 PM
6 points
0
in reply to: xpym’s comment on: What are the strongest arguments for very short timelines?
Is this the consensus view? I think it’s generally agreed that software development has been sped up. A factor of two is ambitious! But that’s what it seems to me, and I’ve measured three examples of computer vision programming, each taking an hour or two, by doing them by hand and then with machine assistance. The machines are dumb and produce results that require rewriting. But my code is also inaccurate on a first try. I don’t have any references where people agree with me. And this may not apply to AI programming in general.

You ask about “anonymous reports of diminishing returns to scaling.” I have also heard these reports, direct from a friend who is a researcher inside a major lab. But note that this does not imply a diminished rate of progress, since there are other ways to advance besides making LLMs bigger. O1 and o3 indicate the payoffs to be had by doing things other than pure scaling. If there are forms of progress available to cleverness, then the speed of advance need not require scaling.

Carl Feynman Dec 26, 2024, 5:07 PM
4 points
0
on: What are the main arguments against AGI?
One argument against is that I think it’s coming soon, and I have a 40 year history of frothing technological enthusiasm, often predicting things will arrive decades before they actually do. 😀

Carl Feynman Dec 26, 2024, 5:03 PM
9 points
2
in reply to: lc’s comment on: lc’s Shortform
These criticisms are often made of “market dominant minorities”, to use a sociologist’s term for what American Jews and Indian-Americans have in common. Here’s a good short article on the topic: https://scholarship.law.duke.edu/cgi/viewcontent.cgi?article=5582&context=faculty_scholarship

Carl Feynman Dec 26, 2024, 4:47 PM
9 points
0
in reply to: Purplehermann’s comment on: Purplehermann’s Shortform
This isn’t crazy— people have tried related techniques. But it needs more details thought out.

In the chess example, the AIs start out very stupid, being wired at random. But in a game between two idiots, moving at random, eventually someone is going to win. And then you reinforce the techniques used by the winner, and de-reinforce the ones used by the loser. In any encounter, you learn, regardless of who wins. But in an encounter between a PM and a programmer, if the programmer fails, who gets reinforced? It might be because the programmer is dumb, and should be de-reinforced. But it might be because the PM is dumb, and asked for something impossible or far beyond what can be done, in which case it should be de-reinforced. But it might be because the PM came up with a task just barely beyond the programmer’s ability, which is good and should be reinforced. We somehow need to keep the PM producing problems which are hard but possible. Maybe the programmer could be tasked with coming up with either a solution or a proof of impossibility?

AlphaGo had a mechanism which tracked how important each move was. It was trained to predict the probability that white would win, on each position encountered in the game. Moves where this probability swung wildly were given a larger weight in reinforcement. This was important for concentrating training on decisive moves, allowing the extraction of information from each move instead of each game. It’s not clear if this is possible in the programming task.

Carl Feynman Dec 26, 2024, 3:19 PM
6 points
0
on: Why is neuron count of human brain relevant to AI timelines?
This is a great question!

Point one:
The computational capacity of the brain used to matter much more than it matters now. The AIs we have now are near-human or superhuman at many skills, and we can measure how skill capacity varies with resources in the near-human range. We can debate and extrapolate and argue with real data.
But we spent decades where the only intelligent system we had was the human brain, so it was the only anchor we had for timelines. So even though it’s very hard to make good estimates from, we had to use it.
Point two:
Most information that gives rise to the human mind is learned, not evolved.
The information encoded by evolution is less than a hundred megabytes. It’s limited by the size of the genome (1 gigabytes). Moreover, we know that much of the genome is unimportant for mental development. About 40% is parasitic (viruses and transposons). Much of the remaining DNA is not under evolutionary control, varying randomly between individuals. Of expressed genes, only about a quarter appear to be expressed in the brain. And some of them encode things AI doesn’t need, like the high-reliability plumbing of the circle of Willis, or the mysteries of love, or the biochemical pickiness of the blood-brain barrier, or wanting to pee when you hear running water. So the “program” contributed by evolution is no more than the size of a largish program like a compiler. (I would claim it’s probably even less. I think the important instincts plus the learning algorithms are only a few thousand lines of code. But that’s debatable.)
On the other hand, the amount learned in a lifetime is on the order of one or a few gigabytes.
Point three:
Most of the information accumulated by evolution has been destroyed. All of the information accumulated in a species is lost when that species goes extinct. And most species have gone extinct, leaving no descendants. The world of the Permian period was (as far as we know) just as busy as today, with hundreds of thousands of animal species. Just one of those species, a little burrowing critter among many other types of little burrowing critters, was the ancestor of all mammals. All the other little burrowing critters lost out. All their evolutionary innovations have been lost.
This doesn’t apply to species with horizontal transmission of genes, like bacteria. But it applies to animals, who are the only creatures with brains.

Carl Feynman Dec 26, 2024, 2:23 PM
2 points
0
in reply to: Purplehermann’s comment on: Purplehermann’s Shortform
What’s a PM?

Carl Feynman Dec 24, 2024, 10:34 PM
10 points
0
in reply to: Steven Byrnes’s comment on: What are the strongest arguments for very short timelines?
I disagree that there is a difference of kind between “engineering ingenuity” and “scientific discovery”, at least in the business of AI. The examples you give—self-play, MCTS, ConvNets—were all used in game-playing programs before AlphaGo. The trick of AlphaGo was to combine them, and then discover that it worked astonishingly well. It was very clever and tasteful engineering to combine them, but only a breakthrough in retrospect. And the people that developed them each earlier, for their independent purposes? They were part of the ordinary cycle of engineering development: “Look at a problem, think as hard as you can, come up with something, try it, publish the results.” They’re just the ones you remember, because they were good.
Paradigm shifts do happen, but I don’t think we need them between here and AGI.

Carl Feynman Dec 24, 2024, 9:56 PM
13 points
2
in reply to: Radford Neal’s comment on: What are the strongest arguments for very short timelines?
Came here to say this, got beaten to it by Radford Neal himself, wow! Well, I’m gonna comment anyway, even though it’s mostly been said.
Gallagher proposed belief propagation as an approximate good-enough method of decoding a certain error-correcting code, but didn’t notice that it worked on all sorts of probability problems. Pearl proposed it as a general mechanism for dealing with probability problems, but wanted perfect mathematical correctness, so confined himself to tree-shaped problems. It was their common generalization that was the real breakthrough: an approximate good-enough solution to all sorts of problems. Which is what Pearl eventually noticed, so props to him.
If we’d had AGI in the 1960s, someone with a probability problem could have said “Here’s my problem. For every paper in the literature, spawn an instance to read that paper and tell me if it has any help for my problem.” It would have found Gallagher’s paper and said “Maybe you could use this?”

Carl Feynman Dec 24, 2024, 9:23 PM
6 points
1
on: What are the strongest arguments for very short timelines?
Summary: Superintelligence in January-August, 2026. Paradise or mass death, shortly thereafter.
This is the shortest timeline proposed in these answers so far. My estimate (guess) is that there’s only 20% of this coming true, but it looks feasible as of now. I can’t honestly assert it as fact, but I will say it is possible.
It’s a standard intelligence explosion scenario: with only human effort, the capacities of our AIs double every two years. Once AI gets good enough to do half the work, we double every one year. Once we’ve done that for a year, our now double-smart AIs help us double in six months. Then we double in three months, then six weeks.… to perfect ASI software, running at the the limits of our hardware, in a finite time. Then the ASI does what it wants, and we suffer what we must.
I hear you say “Carl, this argument is as old as the hills. It hasn’t ever come true, why bring it up now?” The answer is, I bring it up because it seems to be happening.
- For at least six months now, we’ve had software assistants that can roughly double the productivity of software development. At least at the software company where I work, people using AI (including me) are seeing huge increases in effectiveness. I assume the same is true of AI software development, and that AI labs are using this technology as much as they can.
- In the last few months, there’s been a perceptible increase in the speed of releases of better models. I think this is a visible (subjective, debatable) sign of an intelligence explosion starting up.
So I think we’re somewhere in the “doubling in one year” phase of the explosion. If we’re halfway through that year, the singularity is due in August 2026. If we’re near the end of that year, the date is January 2026.
There are lots of things that might go wrong with this scenario, and thereby delay the intelligence explosion. I will mention a few, so you don’t have to.
First, the government might stop the explosion, by banning AI being used for the development of AI. Or perhaps the management of all major AI labs will spontaneously not be so foolish as to. This will delay the problem for an unknown time.
Second, the scenario has an extremely naive model of intelligence explosion microeconomics. It assumes that one doubling of “smartness” produces one doubling of speed. In Yudkowsky’s original scenario, AIs were doing all the work of development, and this might be a sensible assumption. But what has actually happened is that successive generations of AI can handle larger and larger tasks, before they go off the rails. And they can handle these tasks far faster than humans. So the way we work now is that we ask the AI to do some small task, and bang, it’s done. It seems like testing is showing that current AIs can do things that would take a human up to an hour or two. Perhaps the next generation will be able to do tasks up to four hours. The model assumes that this allows a twofold speedup, then fourfold, etc. But this assumption is unsupported.
Third, the scenario assumes that near-term hardware is sufficient for superintelligence. There isn’t time for the accelerating loop to take effect in hardware. Even if design was instant, the physical processes of mask making, lithography, testing, yield optimization and mass production take more than a year. The chips that the ASI will run on in mid-2026 have their design almost done now, at the end of 2024. So we won’t be able to get to ASI, if the ASI requires many orders of magnitude more FLOPs than current models. Instead, we’ll have to wait until the AI designs future generations of semiconductor technology. This will delay matters by years (if using humans to build things) or hours (if using nanotechnology.)
(I don’t think the hardware limit is actually much of a problem; AIs have recently stopped scaling in numbers of parameters and size of training data. Good engineers are constantly figuring out how to pack more intelligence into the same amount of computation. And the human brain provides an existence proof that human-level intelligence requires much less training data. Like Mr. Helm-Burger above, I think human-equivalent cognition is around 10^15 Flops. But reasonable people disagree with me.)

Carl Feynman Dec 22, 2024, 4:50 PM
2 points
0
on: Robbin’s Farm Sledding Route
There’s a shorter hill with a good slope in McLellan park, about a mile away. It debouches into a flat area, so you can coast a long time and don’t have to worry about hitting a fence. If you’ve got the nerve, you can sled onto a frozen pond and really go far.

The shorter hill means it’s quicker to climb, so it provides roughly equal fun per hour.