Holden Karnofsky

Holden Karnofsky is a co-founder of effective altruism, and pioneered GiveWell.

He was the CEO of Open Philanthropy, but left in April 2024 to become a Visiting Scholar at the Carnegie Endowment for International Peace, where he’s working on international security risks from advances in artificial intelligence.

In May 2012, Karnofsky posted Thoughts on the Singularity Institute (SI), which became the most-upvoted article ever on Less Wrong. It offered a detailed critique of what is now the Machine Intelligence Research Institute, and spawned a great deal of discussion.

MIRI staff posted two replies:

• Eliezer Yudkowsky, Reply to Holden on ‘Tool AI’

• Luke Muehlhauser, Reply to Holden on The Singularity Institute

Paul Crowley (“ciphergoth”) posted discussion articles for each point raised:

• Objection 1: it seems to me that any AGI that was set to maximize a “Friendly” utility function would be extraordinarily dangerous.

• Objection 2: SI appears to neglect the potentially important distinction between “tool” and “agent” AI.

• Objection 3: SI’s envisioned scenario is far more specific and conjunctive than it appears at first glance, and I believe this scenario to be highly unlikely.

• Is SI the kind of organization we want to bet on?

• Other objections to SI’s views

Other discussion:

• Phil Goetz, Holden’s Objection 1: Friendliness is dangerous

Links and related pages

• Holden Karnofsky’s posts on LessWrong

• Tool AI