It’s correct that there’s a distinction between whether people identify as pessimistic and whether they are pessimistic in their outlook. I think the first claim is false, and I actually also think the second claim is false, though I am less confident in that.
Rohin reported an unusually large (90%) chance that AI systems will be safe without additional intervention. His optimism was largely based on his belief that AI development will be relatively gradual and AI researchers will correct safety issues that come up.
...without AI alignment, AI systems are reasonably likely to cause an irreversible catastrophe like human extinction. I think most people can agree that this would be bad, though there’s a lot of reasonable debate about whether it’s likely. I believe the total risk is around 10–20%, which is high enough to obsess over.
I go back and forth more than I can really justify, but if you force me to give an estimate it’s probably around 33%; I think it’s very plausible that we die, but more likely that we survive (at least for a little while).
Step 1: sort out our fundamental confusions about agency
Step 2: ambitious value learning (i.e. build an AI which correctly learns human values and optimizes for them)
Step 3: …
Step 4: profit!
… and do all that before AGI kills us all.
That sounds… awfully optimistic. Do you actually think that’s viable?
Better than a 50⁄50 chance of working in time.
Davidad also feels to me like an optimist to me about the world — someone who is excited about solving the problems and finding ways to win, and is excited about other people and ready to back major projects to set things on a good course. I don’t know his probability of an AI takeover but I stand by that he doesn’t seem pessimistic in personality.
On occasion when talking to researchers, I talk to someone who is optimistic that their research path will actually work. I won’t name who but I recently spoke with a long-time researcher who believes that they have a major breakthrough and will be able to solve alignment. I think researchers can trick themselves into thinking they have a breakthrough when they don’t, and this field is unusually lacking in feedback, so I’m not saying I straightforwardly buy their claims, but I think it’s inaccurate to describe them all as pessimistic.
One story we could tell is that the thing these people have in common is that they take alignment seriously, not that they are generally pessimists.
I think alignment is unsolved in the general case and so this makes it harder to strongly argue that it will get solved for future systems, but I don’t buy that people would not update on seeing a solution or strong arguments for that conclusion, and I think that some of Quintin’s and Nora’s arguments have caused people I know to rethink their positions and update some in that direction.
I think the rationalist and EA spaces have been healthy enough for people to express quite extreme positions of expecting an AI-takeover-slash-extinction. I think it would be a strongly negative sign for everyone in these spaces to have identical views or for everyone to give up all hope on civilization’s prospects; but in the absence of that I think it’s a sign of health that people are able to be open about having very strong views. I also think the people who most confidently anticipate an AI takeover sometimes feel and express hope.
I don’t think everyone is starting with pessimism as their bottom line, and I think it’s inaccurate to describe the majority of people in these ecosystems as temperamentally pessimistic or epistemically pessimistic.
It’s correct that there’s a distinction between whether people identify as pessimistic and whether they are pessimistic in their outlook. I think the first claim is false, and I actually also think the second claim is false, though I am less confident in that.
Interview with Rohin Shah in Dec ’19
Paul Christiano in Dec ’22
Scott Alexander, in Why I Am Not (As Much Of) A Doomer (As Some People) in March ’23
John Wentworth in Dec ’21 (also see his to-me-inspiring stump speech from a month later):
Davidad also feels to me like an optimist to me about the world — someone who is excited about solving the problems and finding ways to win, and is excited about other people and ready to back major projects to set things on a good course. I don’t know his probability of an AI takeover but I stand by that he doesn’t seem pessimistic in personality.
On occasion when talking to researchers, I talk to someone who is optimistic that their research path will actually work. I won’t name who but I recently spoke with a long-time researcher who believes that they have a major breakthrough and will be able to solve alignment. I think researchers can trick themselves into thinking they have a breakthrough when they don’t, and this field is unusually lacking in feedback, so I’m not saying I straightforwardly buy their claims, but I think it’s inaccurate to describe them all as pessimistic.
A few related thoughts:
One story we could tell is that the thing these people have in common is that they take alignment seriously, not that they are generally pessimists.
I think alignment is unsolved in the general case and so this makes it harder to strongly argue that it will get solved for future systems, but I don’t buy that people would not update on seeing a solution or strong arguments for that conclusion, and I think that some of Quintin’s and Nora’s arguments have caused people I know to rethink their positions and update some in that direction.
I think the rationalist and EA spaces have been healthy enough for people to express quite extreme positions of expecting an AI-takeover-slash-extinction. I think it would be a strongly negative sign for everyone in these spaces to have identical views or for everyone to give up all hope on civilization’s prospects; but in the absence of that I think it’s a sign of health that people are able to be open about having very strong views. I also think the people who most confidently anticipate an AI takeover sometimes feel and express hope.
I don’t think everyone is starting with pessimism as their bottom line, and I think it’s inaccurate to describe the majority of people in these ecosystems as temperamentally pessimistic or epistemically pessimistic.