This letter was an important milestone in the evolution of MIRI’s strategy over 2020-2024. As of October 2023 Yudkowsky is MIRI’s chair and “the de facto reality (is) that his views get a large weight in MIRI strategic direction”.
MIRI used to favor technical alignment over policy work. In April 2021, in comments to Death with Dignity Yudkowsky argued that:
How about if you solve a ban on gain-of-function research first, and then move on to much harder problems like AGI? A victory on this relatively easy case would result in a lot of valuable gained experience, or, alternatively, allow foolish optimists to have their dangerous optimism broken over shorter time horizons.
You seem confused about my exact past position. I was arguing against EAs who were like, “We’ll solve AGI with policy, therefore no doom.” I am not presently a great optimist about the likelihood of policy being an easy solution. There is just nothing else left.
You’re reading too much into this review. It’s not about your exact position in April 2021, it’s about the evolution of MIRI’s strategy over 2020-2024, and placing this Time letter in that context. I quoted you to give a flavor of MIRI attitudes in 2021 and deliberately didn’t comment on it to allow readers to draw their own conclusions.
I could have linked MIRI’s 2020 Updates and Strategy, which doesn’t mention AI policy at all. A bit dull.
In September 2021, there was a Discussion with Eliezer Yudkowsky which seems relevant. Again, I’ll let readers draw their own conclusions, but here’s a fun quote:
I wasn’t really considering the counterfactual where humanity had a collective telepathic hivemind? I mean, I’ve written fiction about a world coordinated enough that they managed to shut down all progress in their computing industry and only manufacture powerful computers in a single worldwide hidden base, but Earth was never going to go down that route. Relative to remotely plausible levels of future coordination, we have a technical problem.
I welcome deconfusion about your past positions, but I don’t think they’re especially mysterious.
I was arguing against EAs who were like, “We’ll solve AGI with policy, therefore no doom.”
The thread was started by Grant Demaree, and you were replying to a comment by him. You seem confused about Demaree’s exact past position. He wrote, for example: “Eliezer gives alignment a 0% chance of succeeding. I think policy, if tried seriously, has >50%”. Perhaps this is foolish, dangerous, optimism. But it’s not “no doom”.
This letter was an important milestone in the evolution of MIRI’s strategy over 2020-2024. As of October 2023 Yudkowsky is MIRI’s chair and “the de facto reality (is) that his views get a large weight in MIRI strategic direction”.
MIRI used to favor technical alignment over policy work. In April 2021, in comments to Death with Dignity Yudkowsky argued that:
People were not completely swayed by this advice, and the Best of LessWrong 2022 included What an Actually Pessimistic Containment Strategy Looks Like in April 2022 and Let’s Think About Slowing Down AI in December 2022.
In this Time letter from March 2023 we see Yudkowsky doing AI policy work. In January 2024 the new MIRI CEO announced in the MIRI 2024 Mission and Strategy Update that AI policy work provides “a glimmer of hope”. In December 2024 we get Communications in Hard Mode—My New Job at MIRI.
A question for all of us: “How could I have thought that faster?”
You seem confused about my exact past position. I was arguing against EAs who were like, “We’ll solve AGI with policy, therefore no doom.” I am not presently a great optimist about the likelihood of policy being an easy solution. There is just nothing else left.
You’re reading too much into this review. It’s not about your exact position in April 2021, it’s about the evolution of MIRI’s strategy over 2020-2024, and placing this Time letter in that context. I quoted you to give a flavor of MIRI attitudes in 2021 and deliberately didn’t comment on it to allow readers to draw their own conclusions.
I could have linked MIRI’s 2020 Updates and Strategy, which doesn’t mention AI policy at all. A bit dull.
In September 2021, there was a Discussion with Eliezer Yudkowsky which seems relevant. Again, I’ll let readers draw their own conclusions, but here’s a fun quote:
I welcome deconfusion about your past positions, but I don’t think they’re especially mysterious.
The thread was started by Grant Demaree, and you were replying to a comment by him. You seem confused about Demaree’s exact past position. He wrote, for example: “Eliezer gives alignment a 0% chance of succeeding. I think policy, if tried seriously, has >50%”. Perhaps this is foolish, dangerous, optimism. But it’s not “no doom”.