Ben Pace

• Ben PaceFeb 3, 2025, 7:37 PM

  2 points

  0

  in reply to: Ben Pace’s comment on: The Failed Strat­egy of Ar­tifi­cial In­tel­li­gence Doomers

  FWIW in my mind I was comparing this to things like Glen Weyl’s Why I Am Not a Technocrat, and thought this was much better. (Related: Scott Alexander’s response, Weyl’s counter-response).

• Ben PaceFeb 1, 2025, 6:13 AM

  4 points

  −2

  on: The Failed Strat­egy of Ar­tifi­cial In­tel­li­gence Doomers

  I wrote that this “is the best sociological account of the AI x-risk reduction efforts of the last ~decade that I’ve seen.” The line has some disagree reacts inline; I expect this is primarily an expression that the disagree-ers have a low quality assessment of the article, but I would be curious to see links to any other articles or posts that attempt something similar to this one, in order to compare whether they do better/​worse/​different. I actually can’t easily think of any (which is why I felt it was not that bold to say this was the best).

  Edit: I’ve expanded the opening paragraph, to not confuse my comment for me agreeing with the object level assessment of the article..

• Ben PaceFeb 1, 2025, 6:07 AM

  16 points

  1

  in reply to: aysja’s comment on: The Failed Strat­egy of Ar­tifi­cial In­tel­li­gence Doomers

  I’m not particularly resolute on this question. But I get this sense when I look at (a) the best agent foundations work that’s happened over ~10 years of work on the matter, and (b) the work output of scaling up the number of people working on ‘alignment’ by ~100x.

  For the first, trying to get a better understand of the basic concepts like logical induction and corrigibility and low-impact and ontological updates, while I feel like there’s been progress (in timeless decision theory taking a clear step forward in figuring out how think about decision-makers as algorithms; in logical induction as moving forward on how to think about logical uncertainty; notably in the Embedded Agency sequence outlining many basic confusions; and in various writings like Radical Probabilism and Geometric Rationality in finding the breaking edges of expected utility maximization) I don’t feel like the work done over the last 10 years is on track to be a clear ~10% of the work needed.

  I’m not confident it makes sense to try to count it linearly. But I don’t know that there’s enough edges here or new results to feel good about, given 10x as much time to think about it, a new paradigm /​ set of concepts falling into place.

  For the second, I think mostly there’s been (as Wentworth would say) a lot of street-lighting, and a lot of avoiding of actually working on the problem. I mean, there’s definitely been a great amount of bias introduced by ML labs having billions of dollars and setting incentives, but I don’t feel confident that good things would happen in the absence of that. I’d guess that most ideas for straightforwardly increasing the number of people working on these problems will result in them bouncing off and doing unrelated things.

  I think partly I’m also thinking that very few researchers cared about these problems in the last few decades before AGI seemed like a big deal, and still very few researchers seem to care about them, and when I’ve see researchers like Bengio and Sutskever talk about it’s looked to me like they bounce off /​ become very confident they’ve solved the problems while missing obvious things, so my sense is that it will continue to be a major uphill battle to get the real problems actually worked on.

  Perhaps I should focus on a world where I get to build such a field and scale it slowly and set a lot of the culture. I’m not exactly sure how ideal of a setup I should be imagining. Given 100 years, I would give it my best shot. My gut right now says I’d have maybe a 25% chance of success, though if I have to deal with as much random bullshit as we have so far in this timeline (random example: my CEO being unable to do much leadership of Lightcone due to 9 months of litigation from the FTX fallout) then I am less confident.

  My guess is that given 100 years I would be slightly more excited to try out the human intelligence enhancement storyline. But I’ve not thought about that one much, I might well update against it as I learn more of the details.

• Ben PaceFeb 1, 2025, 3:40 AM

  6 points

  3

  in reply to: samuelshadrach’s comment on: The Failed Strat­egy of Ar­tifi­cial In­tel­li­gence Doomers

  Can I double-click on what “does not understand politics at [a] very deep level” means? Can someone explain what they have in mind? I think Eliezer has probably better models than most of what our political institutions are capable of, and probably isn’t very skilled at personally politicking. I’m not sure what other people have in mind.

• Ben PaceFeb 1, 2025, 3:38 AM

  2 points

  0

  in reply to: yams’s comment on: The Failed Strat­egy of Ar­tifi­cial In­tel­li­gence Doomers

  The former, but the latter is a valid response too.

  Someone doing a good job of painting an overall picture is a good opportunity to reflect on the overall picture and what changes to make, or what counter-arguments to present to this account.

• Ben PaceFeb 1, 2025, 3:36 AM

  26 points

  3

  in reply to: jimrandomh’s comment on: The Failed Strat­egy of Ar­tifi­cial In­tel­li­gence Doomers

  For what it’s worth, I have grown pessimistic about our ability to solve the open technical problems even given 100 years of work on them. I think it possible but not probable in most plausible scenarios.

  Correspondingly the importance I assign to increasing the intelligence of humans has drastically increased.

The Failed Strat­egy of Ar­tifi­cial In­tel­li­gence Doomers

Thread for Sense-Mak­ing on Re­cent Mur­ders and How to Sanely Respond

Lighthaven Se­quences Read­ing Group #20 (Tues­day 02/​04)

• Ben PaceJan 28, 2025, 4:26 PM

  2 points

  0

  in reply to: Quinn’s comment on: Ben Pace’s Short­form Feed

  My feelings here aren’t at all related to any news or current events. I could’ve written this any time in the last year or two.

• Ben PaceJan 27, 2025, 2:22 AM

  3 points

  0

  in reply to: M. Y. Zuo’s comment on: Ben Pace’s Short­form Feed

  Can you give me your best one-or-two-line guess? I think the question is trivial from what I’ve written and I don’t really know why you’re not also finding it clear.

• Ben PaceJan 27, 2025, 1:37 AM

  2 points

  0

  in reply to: M. Y. Zuo’s comment on: Ben Pace’s Short­form Feed

  For over a decade I have examined the evidence, thought about the situation from many different perspectives (political, mathematical, personal, economic, etc), and considered arguments and counterarguments. This is my honest understanding of the situation, and I am expressing how I truly feel about that.

• Ben PaceJan 26, 2025, 10:06 PM

  8 points

  0

  on: A Three-Layer Model of LLM Psychology

  Curated. Thanks for writing this! I don’t believe the ideas in this post are entirely original (e.g. character /​ ground is similar to the distinction between simulator /​ simulacra), but I’m going to keep repeating that it’s pro-social to present a good idea in lots of different ways, and indeed reading this post has helped it fit together better in my mind.

• Ben PaceJan 26, 2025, 10:05 PM

  11 points

  3

  on: The Case Against AI Con­trol Research

  Curated! I think this is a fantastic contribution to the public discourse about AI control research. This really helped me think concretely about the development of AI and the likely causes of failure. I also really got a lot out of the visualization at the end of the “Failure to Generalize” section in terms of trying to understand why an AI’s cognition will be alien and hard to interpret. In my view there are already quite a lot of high-level alien forces running on humans (e.g. Moloch), and there will be high-level alien forces running on the simulated society in the AI’s mind.

  I am glad that there’s a high-quality case for and against this line of research, it makes me feel positive about the state of discourse on this subject.

  (Meta note: This curation notice accidentally went up 3 days after the post was curated.)

• Ben PaceJan 26, 2025, 1:31 AM

  3 points

  0

  in reply to: gwern’s comment on: Learn­ing By Writing

  However it is on his LinkedIn.

• Ben PaceJan 26, 2025, 12:07 AM

  4 points

  1

  in reply to: tailcalled’s comment on: Ben Pace’s Short­form Feed

  Yes; she has come to visit me for two months, and I have helped her get into a daily writing routine while she’s here. I know she has the ability to finish at least one.

Lighthaven Se­quences Read­ing Group #19 (Tues­day 01/​28)

• Ben PaceJan 25, 2025, 8:57 AM

  13 points

  1

  in reply to: testingthewaters’s comment on: Ben Pace’s Short­form Feed

  Thank you.

  It does not currently look to me like we will win this war, speaking figuratively. But regardless, I still have many opportunities to bring truth, courage, justice, honor, love, playfulness, and other virtues into the world, and I am a person whose motivations run more on living out virtues rather than moving toward concrete hopes. I will still be here building things I love, like LessWrong and Lighthaven, until the end.

• Ben PaceJan 25, 2025, 3:59 AM

  36 points

  −5

  on: Ben Pace’s Short­form Feed

  So many people have lived such grand lives. I have certainly lived a greater life than I expected, filled with adventures and curious people. But people will soon not live any lives at all. I believe that we will soon build intelligences more powerful than us who will disempower and kill us all. I will see no children of mine grow to adulthood. No people will walk through mountains and trees. No conscious mind will discover any new laws of physics. My mother will not write all of the novels she wants to write. The greatest films that will be made have probably been made. I have not often viscerally reflected on how much love and excitement I have for all the things I could do in the future, so I didn’t viscerally feeling the loss. But now, when it is all lost, I start to think on it. And I just want to weep. I want to scream and smash things. Then I just want to sit quietly and watch the sun set, with people I love.

• Ben PaceJan 24, 2025, 6:21 PM

  4 points

  0

  in reply to: ryan_greenblatt’s comment on: An­nounc­ing Dialogues

  I am sad they’re not getting as much use. I have wondered if they would work well as part of the comment section UI, where if you’re having a back-and-forth with someone, the site instead offers you “Would you like to have a dialogue instead?” with a single button.