Positively transformative AI systems could reduce the overall risk from AI by: preventing the construction of a more dangerous AI; changing something about how global governance works; instituting surveillance or oversight mechanisms widely; rapidly and safely performing alignment research or other kinds of technical research; greatly improving cyberdefense; persuasively exposing misaligned behaviour in other AIs and demonstrating alignment solutions, and through many other actions that incrementally reduce risk.
One common way of imagining this process is that an aligned AI could perform a ‘pivotal act’ that solves AI existential safety in one swift stroke. However, it is important to consider this much wider range of ways in which one or several transformative AI systems could reduce the total risk from unaligned transformative AI.
Is it important to consider the wide range of ways in which a chimp could beat Garry Kasparov in a single chess match, or the wide range of ways in which your father [or, for that matter, von Neumann] could beat the house after going to Vegas?
Sorry if I sound arrogant, but this is a serious question. Sometimes differences in perspective can be large enough to warrant asking such silly-sounding questions.
I am unclear where you think the problem for a superintelligence [which is smart enough to complete some technologically superhuman pivotal act] is non-vanishingly-likely to come in, from a bunch of strictly less smart beings which existed previously, and which the smarter ASI can fully observe and outmaneuver.
If you don’t think the intelligence difference is likely to be big enough that “the smarter ASI can fully observe and outmaneuver” the previously-extant, otherwise-impeding thinkers, then I understand where our difference of opinion lies, and would be happy to make my extended factual case that that’s not true.
The point is that in this scenario you have aligned AGI or ASI on your side. On the assumption that the other side has/is a superintelligence and you are not, then yes this is likely a silly question, but I talk about ‘TAI systems reducing the total risk from unaligned TAI’. So this is the chimp with access to a chess computer playing Gary Kasparov at chess.
And to be clear, on any slower takeoff scenario where there’s intermediate steps from AGI to ASI, the analogy might not be quite that. In the scenario where there I’m talking about multiple actions, I’m usually talking about a moderate or slow takeoff where there are intermediately powerful AIs and not just a jump to ASI.
Yes, for me, though I’d give this low probability mostly due to the chimp breaking the computer, and most of the problems for the chimp fundamentally come down to their body structure being very unoptimized for tools, rather than them being less intelligent absolutely, and humans have much better abilities to use tools than chimps.
I’m inclined to conclude from this that you model the gulf between chimp society and human society as in general having mostly anatomical rather than cognitive causes. Is that correct?
I’d say that cognitive causes like coordination/pure smartness do matter, but yes I’m non-trivially stating that a big cause of the divergence between chimps and humans is because of their differing anatomies, combined with an overhang from evolution where evolution spent it’s compute shockingly inefficiently compared to us, though we haven’t reduced the overhang to 0, and some exploitable overhangs from evolution still remain.
I’m modeling this as multiple moderate influences, plus a very large influenced added up to a ridiculous divergence.
AFAIK, there’s around as much evidence for non-human ape capability in the mirror test as there is for certain cetaceans and magpies, and the evidence on apes being somewhat capable of syntactical sign language is mixed to dubious. It’s true the internal politics of chimp societies are complex and chimps have been known to manipulate humans very complexly for rewards, but on both counts the same could be said of many other species [corvids, cetaceans, maybe elephants] none of which I would put on approximate par with humans intelligence-wise, in the sense of crediting them with plausibly being viable halves of grandmaster-level chess centaurs.
I’m curious if you have an alternate perspective wrt any of these factors, or if your divergence from me here comes from looking for intelligence in a different place.
Also, I’m not sure what you mean by “overhang [from] evolution spen[ding] its compute shockingly inefficiently” in this context.
My alternate perspective here is that while IQ/intelligence actually matters, I don’t think that the difference is so large as to explain why chimp society was completely outclassed, and I usually model mammals as having 2 OOMs worse to a few times better intelligence than us, though usually towards the worse end of that range, so other factors matter.
So the difference is I’m less extreme than this:
none of which I would put on approximate par with humans intelligence-wise.
On the overhang from evolution spending it’s compute shockingly inefficiently, I was referring to this post, where evolution was way weaker than in-lifetime updating, for the purposes of optimization:
Is it important to consider the wide range of ways in which a chimp could beat Garry Kasparov in a single chess match, or the wide range of ways in which your father [or, for that matter, von Neumann] could beat the house after going to Vegas?
Sorry if I sound arrogant, but this is a serious question. Sometimes differences in perspective can be large enough to warrant asking such silly-sounding questions.
I am unclear where you think the problem for a superintelligence [which is smart enough to complete some technologically superhuman pivotal act] is non-vanishingly-likely to come in, from a bunch of strictly less smart beings which existed previously, and which the smarter ASI can fully observe and outmaneuver.
If you don’t think the intelligence difference is likely to be big enough that “the smarter ASI can fully observe and outmaneuver” the previously-extant, otherwise-impeding thinkers, then I understand where our difference of opinion lies, and would be happy to make my extended factual case that that’s not true.
The point is that in this scenario you have aligned AGI or ASI on your side. On the assumption that the other side has/is a superintelligence and you are not, then yes this is likely a silly question, but I talk about ‘TAI systems reducing the total risk from unaligned TAI’. So this is the chimp with access to a chess computer playing Gary Kasparov at chess.
And to be clear, on any slower takeoff scenario where there’s intermediate steps from AGI to ASI, the analogy might not be quite that. In the scenario where there I’m talking about multiple actions, I’m usually talking about a moderate or slow takeoff where there are intermediately powerful AIs and not just a jump to ASI.
A chimp with a chess computer beating Garry Kasparov seems plausible to you?
Yes, for me, though I’d give this low probability mostly due to the chimp breaking the computer, and most of the problems for the chimp fundamentally come down to their body structure being very unoptimized for tools, rather than them being less intelligent absolutely, and humans have much better abilities to use tools than chimps.
I’m inclined to conclude from this that you model the gulf between chimp society and human society as in general having mostly anatomical rather than cognitive causes. Is that correct?
I’d say that cognitive causes like coordination/pure smartness do matter, but yes I’m non-trivially stating that a big cause of the divergence between chimps and humans is because of their differing anatomies, combined with an overhang from evolution where evolution spent it’s compute shockingly inefficiently compared to us, though we haven’t reduced the overhang to 0, and some exploitable overhangs from evolution still remain.
I’m modeling this as multiple moderate influences, plus a very large influenced added up to a ridiculous divergence.
AFAIK, there’s around as much evidence for non-human ape capability in the mirror test as there is for certain cetaceans and magpies, and the evidence on apes being somewhat capable of syntactical sign language is mixed to dubious. It’s true the internal politics of chimp societies are complex and chimps have been known to manipulate humans very complexly for rewards, but on both counts the same could be said of many other species [corvids, cetaceans, maybe elephants] none of which I would put on approximate par with humans intelligence-wise, in the sense of crediting them with plausibly being viable halves of grandmaster-level chess centaurs.
I’m curious if you have an alternate perspective wrt any of these factors, or if your divergence from me here comes from looking for intelligence in a different place.
Also, I’m not sure what you mean by “overhang [from] evolution spen[ding] its compute shockingly inefficiently” in this context.
My alternate perspective here is that while IQ/intelligence actually matters, I don’t think that the difference is so large as to explain why chimp society was completely outclassed, and I usually model mammals as having 2 OOMs worse to a few times better intelligence than us, though usually towards the worse end of that range, so other factors matter.
So the difference is I’m less extreme than this:
On the overhang from evolution spending it’s compute shockingly inefficiently, I was referring to this post, where evolution was way weaker than in-lifetime updating, for the purposes of optimization:
https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn#Evolution_s_sharp_left_turn_happened_for_evolution_specific_reasons
Thanks for clarifying your thoughts!