The Feeling of Idea Scarcity

johnswentworth31 Dec 2022 17:34 UTC
246 points
22 comments5 min readLW link1 review

Here’s a story you may recognize. There’s a bright up-and-coming young person—let’s call her Alice. Alice has a cool idea. It seems like maybe an important idea, a big idea, an idea which might matter. A new and valuable idea. It’s the first time Alice has come up with a high-potential idea herself, something which she’s never heard in a class or read in a book or what have you.

So Alice goes all-in pursuing this idea. She spends months fleshing it out. Maybe she writes a paper, or starts a blog, or gets a research grant, or starts a company, or whatever, in order to pursue the high-potential idea, bring it to the world.

And sometimes it just works!

… but more often, the high-potential idea doesn’t actually work out. Maybe it turns out to be basically-the-same as something which has already been tried. Maybe it runs into some major barrier, some not-easily-patchable flaw in the idea. Maybe the problem it solves just wasn’t that important in the first place.

From Alice’ point of view, the possibility that her one high-potential idea wasn’t that great after all is painful. The idea probably feels to Alice like the single biggest intellectual achievement of her life. To lose that, to find out that her single greatest intellectual achievement amounts to little or nothing… that hurts to even think about. So most likely, Alice will reflexively look for an out. She’ll look for some excuse to ignore the similar ideas which have already been tried, some reason to think her idea is different. She’ll look for reasons to believe that maybe the major barrier isn’t that much of an issue, or that we Just Don’t Know whether it’s actually an issue and therefore maybe the idea could work after all. She’ll look for reasons why the problem really is important. Maybe she’ll grudgingly acknowledge some shortcomings of the idea, but she’ll give up as little ground as possible at each step, update as slowly as she can.

And this is where a bunch of the standard advice from the sequences comes in. Once you’ve chosen the idea, the only way to improve it is to change the idea or choose a new idea; finding arguments that it’s a good idea will not actually make it better. You won’t make real progress until you give up, say “oops” and move on, so do that quickly rather than slowly. What is true is already so; owning up to it will not make it worse.

… but that advice doesn’t make it much less painful. Trying to forcibly abandon her one high-potential idea can easily leave Alice demotivated, in despair, feeling like she’s failed as a person. Or, Alice notices herself failing to abandon the idea, and that makes her feel like she’s failed as a person. I’ve seen people really tie themselves into knots this way.

An Alternative Path

Alice’ story makes it seem like an emotional attachment to her idea is the main problem, the main thing preventing her from moving on to greener pastures. And that is true, in a sense. But then the obvious response is to directly fight the emotional attachment. And that, I claim, is usually a mistake.

Why a mistake? Because most people do not actually have that level of control over their emotions. “Just fight the emotional attachment” is a plan which pretends Alice can control something which she probably cannot actually control. (A fabricated option, in Duncan’s terminology.) And when it turns out Alice does not have that level of control, she’ll be back where she started, with a whole additional reason to feel like shit.

Worse, it’s a very common mistake to think one has that level of emotional control. Like Hazard’s story:

So young me is upset that the grub master for our camping trip forgot half the food on the menu, and all we have for breakfast is milk. I couldn’t “fix it” given that we were in the woods, so my next option was “stop feeling upset about it.” So I reached around in the dark of my mind, and Oops, the “healthily process feelings” lever is right next to the “stop listening to my emotions” lever.

The end result? “Wow, I decided to stop feeling upset, and then I stopped feeling upset. I’m so fucking good at emotional regulation!!!!!”

So what’s the alternative?

There’s a useful standard mental move where you notice some seemingly “irrational” (i.e. not-accurately-tracking-the-thing-it-claims-to-be-tracking) emotional reaction, then ask what real thing that emotion is tracking. In Alice’ case, her emotional attachment is tracking (perceived) idea scarcity. This is the first seemingly-high-potential idea Alice has had; of course high-potential ideas will feel scarce to her! If she loses this one, then she has no high-potential ideas left.

That suggests an alternative path: rather than viewing emotional attachment as the root problem, view (perceived) idea scarcity as the root problem.

And there’s an obvious way to directly fight idea scarcity, without tying oneself into emotional knots: go explore, and find some other high-potential ideas. Alice doesn’t need to abandon her big idea (yet), she just needs to go find more ideas to complement it and compete with it. If she’s found one, she can probably find others.

(This is similar to Eliezer’s advice to leave a line of retreat, but with the bonus that it requires less explicitly thinking about the world in which Alice’ high-potential idea fails.)

In My Own Life

Way back in high school, I once thought I found an efficient algorithm for integer factorization (a major open problem). I was so excited; it would have been the only major discovery I’d ever made. I remember realizing one step of the logic didn’t work, and telling myself that step seemed unimportant. I remember the feeling of avoiding thinking about it, the instinctive flinch away from the thought of failure and towards reassuring myself that I’d found something big.

The algorithm didn’t work, of course. In that case, fortunately, reality provided a sufficiently unambiguous feedback loop that I couldn’t deceive myself for long. Months, not years.

… that wasn’t enough to learn the pattern, though. Shortly out of college, reality was still hitting me over the head; that time the big idea was an efficient implementation of universal competitively-optimal portfolios. I lost a couple thousand dollars on wildly over-leveraged forex positions. After that I finally started to eye my promising-seeming ideas with some instinctive mistrust. Not stop pursuing them—I learned a lot from even the failed ideas, and over time I became noticeably better at evaluating my own supposedly-big ideas, as well as more systematically searching for big ideas. That instinctive mistrust, that tendency to look for ways my ideas might fail rather than only ways they might work, was a key load-bearing part of becoming better at it.

And I don’t think it’s a coincidence that I had to go through multiple seemingly-big ideas to get there. In hindsight, each time it got easier to abandon the idea when it didn’t work out. It’s not just learning from experience, it’s the gut-level anticipation that there will be more ideas in the future. I don’t have zero emotional attachment to my ideas, of course—my current work on natural abstraction is beautiful and I love it. But if it turns out that my approach to natural abstraction is all totally wrong, or totally useless, I wouldn’t feel like I’ve been reduced to nothing. It’s not the only thread I’m pulling on.

Takeaway

If you want to avoid becoming one of those researchers or entrepreneurs or creators who’s just not emotionally capable of abandoning their supposedly-great idea, even after reality has clearly told them to say “oops” and move on, don’t try to hammer away emotions you don’t actually control. Instead, treat the problem upstream: go find more big ideas. There’s more out there, you can find them. Remove the gut-level feeling that this is your only big idea, your one chance, and you’ll be much better able to see when your idea falls short without feeling like you’ve lost everything.

johnswentworth31 Dec 2022 17:34 UTC
246 points
22 comments5 min readLW link1 review