FWIW, I basically buy all of these, but they are not-at-all sufficient to back up your claims about how superintelligence won’t foom (or whatever your actual intended claims are about takeoff). Insofar as all this is supposed to inform AI threat models, it’s the weakest subclaims necessary to support the foom-claims which are of interest, not the strongest subclaims.
I basically buy all of these, but they are not-at-all sufficient to back up your claims about how superintelligence won’t foom
Foom isn’t something that EY can prove beyond doubt or I can disprove beyond doubt, so this is a matter of subjective priors and posteriors.
If you were convinced of foom inevitability before, these claims are unlikely to convince of the opposite, but they do undermine EY’s argument:
they support the conclusion that the brain is reasonably pareto-efficient (greatly undermining EY’s argument that evolution and the brain are grossly inefficient, as well as his analysis confidence),
they undermine nanotech as a likely source of large FOOM gains, and
weaken EY’s claim of huge software FOOM gains (because the same process which optimized the brain’s hardware platform optimized the wiring/learning algorithms over the same time frame).
The four claims you listed as “central” at the top of this thread don’t even mention the word “brain”, let alone anything about it being pareto-efficient.
It would make this whole discussion a lot less frustrating for me (and probably many others following it) if you would spell out what claims you actually intend to make about brains, nanotech, and FOOM gains, with the qualifiers included. And then I could either say “ok, let’s see how well the arguments back up those claims” or “even if true, those claims don’t actually say much about FOOM because...”, rather than this constant probably-well-intended-but-still-very-annoying jumping between stronger and weaker claims.
Also, I recognize that I’m kinda grouchy about the whole thing and that’s probably coming through in my writing, and I appreciate a lot that you’re responding politely and helpfully on the other side of that. So thankyou for that too.
FWIW, I basically buy all of these, but they are not-at-all sufficient to back up your claims about how superintelligence won’t foom (or whatever your actual intended claims are about takeoff). Insofar as all this is supposed to inform AI threat models, it’s the weakest subclaims necessary to support the foom-claims which are of interest, not the strongest subclaims.
Foom isn’t something that EY can prove beyond doubt or I can disprove beyond doubt, so this is a matter of subjective priors and posteriors.
If you were convinced of foom inevitability before, these claims are unlikely to convince of the opposite, but they do undermine EY’s argument:
they support the conclusion that the brain is reasonably pareto-efficient (greatly undermining EY’s argument that evolution and the brain are grossly inefficient, as well as his analysis confidence),
they undermine nanotech as a likely source of large FOOM gains, and
weaken EY’s claim of huge software FOOM gains (because the same process which optimized the brain’s hardware platform optimized the wiring/learning algorithms over the same time frame).
The four claims you listed as “central” at the top of this thread don’t even mention the word “brain”, let alone anything about it being pareto-efficient.
It would make this whole discussion a lot less frustrating for me (and probably many others following it) if you would spell out what claims you actually intend to make about brains, nanotech, and FOOM gains, with the qualifiers included. And then I could either say “ok, let’s see how well the arguments back up those claims” or “even if true, those claims don’t actually say much about FOOM because...”, rather than this constant probably-well-intended-but-still-very-annoying jumping between stronger and weaker claims.
Ok fair those are more like background ideas/claims, so I reworded that and added 2
Thanks!
Also, I recognize that I’m kinda grouchy about the whole thing and that’s probably coming through in my writing, and I appreciate a lot that you’re responding politely and helpfully on the other side of that. So thankyou for that too.