And (to address something from OP) the checkpoint thing was just the AI being dumb, wasting time and storage space for no good reason. This is veryobviously not a case of “using extra resources” in the sense relevant to instrumental convergence. I’m surprised that this needs pointing out at all, but apparently it does.
I’m not very surprised. I think the broader discourse is very well-predicted by “pessimists[1] rarely (publicly) fact-check arguments for pessimism but demand extreme rigor from arguments for optimism”, which is what you’d expect from standard human biases applied to the humans involved in these discussions.
To illustrate that point, generally it’s the same (apparently optimistic) folk calling out factual errors in doom arguments, even though that fact-checking opportunity is equally available to everyone. Even consider who is reacting “agree” and “hits the mark” to these fact-checking comments—roughly the same story.
Imagine if Eliezer or habryka or gwern or Zvi had made your comment instead, or even LW-reacted as mentioned. I think that’d be evidence of a far healthier discourse.
I’m going to set aside, for the moment, the extent to which there is a symmetric problem with optimists not fact-checking optimist claims. My comment addresses a matter of absolute skill at rationality, not skill relative to the “opposition.”
(This seems false. I have made dozens of similar comments over the years, calling out people making overly doomy or dumb pessimistic predictions. I had also strong-upvoted nostralgebraists comment.
I of course think there are a bunch of dynamics in the space you are pointing to here, but I think the details you point in this comment are wrong. Everyone on the list you mention, me, Gwern, Zvi and Eliezer all frequently call out factual errors in people trying to make bad arguments for pessimism, and indeed I was among the most vocal critics of the AI bioterrorism work. If this is cruxy for you, I could dig up at least 3 examples for each of me, Gwern, Zvi and Eliezer, but would prefer not to spend the time.
Indeed in this very instance I had started writing a similar comment to nostalgebraist’s comments, and then stopped when I saw he basically made the right points.)
I’m not very surprised. I think the broader discourse is very well-predicted by “pessimists[1] rarely (publicly) fact-check arguments for pessimism but demand extreme rigor from arguments for optimism”, which is what you’d expect from standard human biases applied to the humans involved in these discussions.
To illustrate that point, generally it’s the same (apparently optimistic) folk calling out factual errors in doom arguments, even though that fact-checking opportunity is equally available to everyone. Even consider who is reacting “agree” and “hits the mark” to these fact-checking comments—roughly the same story.
Imagine if Eliezer or habryka or gwern or Zvi had made your comment instead, or even LW-reacted as mentioned. I think that’d be evidence of a far healthier discourse.
I’m going to set aside, for the moment, the extent to which there is a symmetric problem with optimists not fact-checking optimist claims. My comment addresses a matter of absolute skill at rationality, not skill relative to the “opposition.”
(This seems false. I have made dozens of similar comments over the years, calling out people making overly doomy or dumb pessimistic predictions. I had also strong-upvoted nostralgebraists comment.
I of course think there are a bunch of dynamics in the space you are pointing to here, but I think the details you point in this comment are wrong. Everyone on the list you mention, me, Gwern, Zvi and Eliezer all frequently call out factual errors in people trying to make bad arguments for pessimism, and indeed I was among the most vocal critics of the AI bioterrorism work. If this is cruxy for you, I could dig up at least 3 examples for each of me, Gwern, Zvi and Eliezer, but would prefer not to spend the time.
Indeed in this very instance I had started writing a similar comment to nostalgebraist’s comments, and then stopped when I saw he basically made the right points.)