My new point is that I don’t think optimal agents will exist when we lose all control
(btw I also realized I didn’t strictly mean ‘optimal’ by ‘superintelligent’, but at least close enough to it / ‘strongly superhuman enough’ for us to not be able to tell the difference. I originally used the ‘optimal’ wording trying to find some other definition apart from ‘super-human’)
it is also plausible to me that life-caring beings first lose control to much narrower programs[1] or moderately superhuman unaligned agents totally outcompeting them economically (if it turns out that making better agents is hard enough that they can’t just directly do that instead), or something.
also, a ‘multipolar AI-driven but still normal-ish’ scenario seems to continue at most until a strong enough agent is created. (e.g. that could be what a race is towards).
(maybe after ‘loss of control to weaker AI’ scenarios, those weaker AIs also keep making better agents afterwards, but i’m not sure about that, because they could be myopic and in some stable pattern/equilibrium)
(btw I also realized I didn’t strictly mean ‘optimal’ by ‘superintelligent’, but at least close enough to it / ‘strongly superhuman enough’ for us to not be able to tell the difference. I originally used the ‘optimal’ wording trying to find some other definition apart from ‘super-human’)
it is also plausible to me that life-caring beings first lose control to much narrower programs[1] or moderately superhuman unaligned agents totally outcompeting them economically (if it turns out that making better agents is hard enough that they can’t just directly do that instead), or something.
also, a ‘multipolar AI-driven but still normal-ish’ scenario seems to continue at most until a strong enough agent is created. (e.g. that could be what a race is towards).
(maybe after ‘loss of control to weaker AI’ scenarios, those weaker AIs also keep making better agents afterwards, but i’m not sure about that, because they could be myopic and in some stable pattern/equilibrium)
(e.g. the ‘going out with a whimper’ part of this post)