Exercise: What mistake is the following sentiment making?
If there’s only a one in a million chance someone can save the world, then there’d better be well more than a million people trying.
Answer:
The whole challenge of “having a one in a million chance of saving the world” is the wrong framing, the challenge is having a positive impact in the first case (for example: by not destroying the world or making things worse, e.g. from s-risks). You could think of this as a setting the zero point thing going on, though I like to think of it in terms of Bayes and Pascel’s wagers:
In terms of Bayes: You’re fixating on the expected value contributed from 10−6(BIG) and ignoring the rest of the 1−10−6 hypothesis space. In most cases, there are corresponding low probability events which “cancel out” the EV contributed from 10−6(BIG)’s direct reasoning.
(I will also note that, empirically, it could be argued Eliezer was massively net-negative from a capabilities advancements perspective; having causal links to founding of deepmind & openai. I bring this up to point out how nontrivial having a positive impact at all is, in a domain like ours)
Exercise: What mistake is the following sentiment making?
Answer:
The whole challenge of “having a one in a million chance of saving the world” is the wrong framing, the challenge is having a positive impact in the first case (for example: by not destroying the world or making things worse, e.g. from s-risks). You could think of this as a setting the zero point thing going on, though I like to think of it in terms of Bayes and Pascel’s wagers:
In terms of Bayes: You’re fixating on the expected value contributed from 10−6(BIG) and ignoring the rest of the 1−10−6 hypothesis space. In most cases, there are corresponding low probability events which “cancel out” the EV contributed from 10−6(BIG)’s direct reasoning.
(I will also note that, empirically, it could be argued Eliezer was massively net-negative from a capabilities advancements perspective; having causal links to founding of deepmind & openai. I bring this up to point out how nontrivial having a positive impact at all is, in a domain like ours)