Superficial stylistic remarks (as you’ll see, I’ve only looked at about the first 1⁄4 of the paper):
The paper repeatedly uses the word “agency” where “agent” would seem more appropriate.
I agree with paper-machine that the mini-biography of I J Good has little value here.
The remark in section 1 about MIRI being funding-limited is out of place and looks like a whine or a plea for more money. Just take it out.
“albeit” on page 10, shortly before footnote 8, should just be “but”. (Or maybe “even though”, if that’s your meaning.) [EDITED to add: there’s another “albeit” that reads oddly to me, in footnote 66 on page 50. It’s not wrong, but it feels odd. Roughly, wherever you can correctly write “albeit” you can equivalently write “even though”, and that’s a funny thing to be starting a footnote with.]
“criteria” in footnote 11 about paperclip maximizers should be “criterion”.
In footnote 15 (about “g”) the word “entrants” seems very weirdly chosen, and the footnote seems to define g as the observed correlation between different measures of intelligence, which is nonsense.
The premise of the paper is that whether or not intelligence explosion will occur is (or at least is being pretended to be) an open question. But at many points within the paper there are references to “the intelligence explosion” that seem to presuppose that there will in fact be one.
Footnote 23 puts “Wikipedia” in italics, which to my eyes looks very strange.
[EDITED to incorporate a comment I made separately, which I’ll now retract.]
Superficial stylistic remarks (as you’ll see, I’ve only looked at about the first 1⁄4 of the paper):
The paper repeatedly uses the word “agency” where “agent” would seem more appropriate.
I agree with paper-machine that the mini-biography of I J Good has little value here.
The remark in section 1 about MIRI being funding-limited is out of place and looks like a whine or a plea for more money. Just take it out.
“albeit” on page 10, shortly before footnote 8, should just be “but”. (Or maybe “even though”, if that’s your meaning.) [EDITED to add: there’s another “albeit” that reads oddly to me, in footnote 66 on page 50. It’s not wrong, but it feels odd. Roughly, wherever you can correctly write “albeit” you can equivalently write “even though”, and that’s a funny thing to be starting a footnote with.]
“criteria” in footnote 11 about paperclip maximizers should be “criterion”.
In footnote 15 (about “g”) the word “entrants” seems very weirdly chosen, and the footnote seems to define g as the observed correlation between different measures of intelligence, which is nonsense.
The premise of the paper is that whether or not intelligence explosion will occur is (or at least is being pretended to be) an open question. But at many points within the paper there are references to “the intelligence explosion” that seem to presuppose that there will in fact be one.
Footnote 23 puts “Wikipedia” in italics, which to my eyes looks very strange.
[EDITED to incorporate a comment I made separately, which I’ll now retract.]
Done.
Done.