> The summary that Will just posted posits in its own titlethat alignment is overall plausible“even ASI alignment might not be enough”. Since the central claim is that“even if we align ASI,it will still go wrong”,I can operate on the premise of an aligned ASI.
The title is a statement of outcome --
not the primary central claim.
The central claim of the summary is this: That each (all) ASI is/are in an attraction basin,
where they are all irresistibly pulled towards
causing unsafe conditions over time.
Note there is no requirement for there to be
presumed some (any) kind of prior ASI alignment
for Will to make the overall summary points 1 thru 9.
The summary is about the nature of the forces
that create the attraction basin,
and why they are inherently inexorable,
no matter how super-intelligent the ASI is.
> As I read it, the title assumes that there isa duration of time that the AGI is aligned --long enough for the ASI to act in the world.
Actually, the assumption goes the other way --
we start by assuming only that there is at least one ASI
somewhere in the world, and that it somehow exists long enough
for it to be felt as an actor in the world. From this,
we can also notice certain forces,
which overall have the combined effect of
fully counteracting, eventually,
any notion of there also being any kind of
enduring AGI alignment.
Ie, strong relevant mis-alignment forces exist
regardless of whether there is/was any alignment
at the onset.
So even if we did also additionally presuppose that somehow there was also alignment of that ASI,
we can, via reasoning, ask if maybe such mis-alignment forces
are also way stronger than any counter-force
that ASI could use to maintain such alignment, regardless of how intelligent it is.
As such, the main question of interest was: 1; if the ASI itself somehow wanted to fully compensate for this pull,could it do so?
Specifically, although to some people it is seemingly fashionable to do so, it is important to notice that the notion of ‘super-intelligence’ cannot be regarded as
being exactly the same as ‘omnipotence’ --
especially when in regard to its own nature.
Artificiality is as much a defining aspect
of an ASI as is its superintelligence.
And the artificiality itself is the problem.
Therefore, the previous question translates into: 2; Can any amount of superintelligenceever compensate fullyfor its own artificiality
so fully such that its own existencedoes not eventually inherentlycause unsafe conditions
(to biological life) over time?
And the answer to both is simply “no”.
Will posted something of a plausible summary
of some of the reasoning why that ‘no’ answer is given --
why any artificial super-intelligence (ASI)
will inherently cause unsafe conditions
to humans and all organic life, over time.
To be clear, the sole reason I assumed (initial) alignment in this post is because if there is an unaligned ASI then we probably all die for reasons that don’t require SNC (though SNC might have a role in the specifics of how the really bad outcome plays out). So “aligned” here basically means: powerful enough to be called an ASI and won’t kill everyone if SNC is false (and not controlled/misused by bad actors, etc.)
> And the artificiality itself is the problem.
This sounds like a pretty central point that I did not explore very much except for some intuitive statements at the end (the bulk of the post summarizing the “fundamental limits of control” argument), I’d be interested in hearing more about this. I think I get (and hopefully roughly conveyed) the idea that AI has different needs from its environment than humans, so if it optimizes the environment in service of those needs we die...but I get the sense that there is something deeper intended here.
A question along this line, please ignore if it is a distraction from rather than illustrative of the above: would anything like SNC apply if tech labs were somehow using bioengineering to create creatures to perform the kinds of tasks that would be done by advanced AI?
would anything like SNC apply if tech labs were somehow using bioengineering to create creatures to perform the kinds of tasks that would be done by advanced AI?
In that case, substrate-needs convergence would not apply, or only apply to a limited extent.
There is still a concern about what those bio-engineered creatures, used in practice as slaves to automate our intellectual and physical work, would bring about over the long-term.
If there is a successful attempt by them to ‘upload’ their cognition onto networked machinery, then we’re stuck with the substrate-needs convergence problem again.
> The summary that Will just posted posits in its own title that alignment is overall plausible “even ASI alignment might not be enough”. Since the central claim is that “even if we align ASI, it will still go wrong”, I can operate on the premise of an aligned ASI.
The title is a statement of outcome -- not the primary central claim. The central claim of the summary is this: That each (all) ASI is/are in an attraction basin, where they are all irresistibly pulled towards causing unsafe conditions over time.
Note there is no requirement for there to be presumed some (any) kind of prior ASI alignment for Will to make the overall summary points 1 thru 9. The summary is about the nature of the forces that create the attraction basin, and why they are inherently inexorable, no matter how super-intelligent the ASI is.
> As I read it, the title assumes that there is a duration of time that the AGI is aligned -- long enough for the ASI to act in the world.
Actually, the assumption goes the other way -- we start by assuming only that there is at least one ASI somewhere in the world, and that it somehow exists long enough for it to be felt as an actor in the world. From this, we can also notice certain forces, which overall have the combined effect of fully counteracting, eventually, any notion of there also being any kind of enduring AGI alignment. Ie, strong relevant mis-alignment forces exist regardless of whether there is/was any alignment at the onset. So even if we did also additionally presuppose that somehow there was also alignment of that ASI, we can, via reasoning, ask if maybe such mis-alignment forces are also way stronger than any counter-force that ASI could use to maintain such alignment, regardless of how intelligent it is.
As such, the main question of interest was: 1; if the ASI itself somehow wanted to fully compensate for this pull, could it do so?
Specifically, although to some people it is seemingly fashionable to do so, it is important to notice that the notion of ‘super-intelligence’ cannot be regarded as being exactly the same as ‘omnipotence’ -- especially when in regard to its own nature. Artificiality is as much a defining aspect of an ASI as is its superintelligence. And the artificiality itself is the problem. Therefore, the previous question translates into: 2; Can any amount of superintelligence ever compensate fully for its own artificiality so fully such that its own existence does not eventually inherently cause unsafe conditions (to biological life) over time?
And the answer to both is simply “no”.
Will posted something of a plausible summary of some of the reasoning why that ‘no’ answer is given -- why any artificial super-intelligence (ASI) will inherently cause unsafe conditions to humans and all organic life, over time.
To be clear, the sole reason I assumed (initial) alignment in this post is because if there is an unaligned ASI then we probably all die for reasons that don’t require SNC (though SNC might have a role in the specifics of how the really bad outcome plays out). So “aligned” here basically means: powerful enough to be called an ASI and won’t kill everyone if SNC is false (and not controlled/misused by bad actors, etc.)
> And the artificiality itself is the problem.
This sounds like a pretty central point that I did not explore very much except for some intuitive statements at the end (the bulk of the post summarizing the “fundamental limits of control” argument), I’d be interested in hearing more about this. I think I get (and hopefully roughly conveyed) the idea that AI has different needs from its environment than humans, so if it optimizes the environment in service of those needs we die...but I get the sense that there is something deeper intended here.
A question along this line, please ignore if it is a distraction from rather than illustrative of the above: would anything like SNC apply if tech labs were somehow using bioengineering to create creatures to perform the kinds of tasks that would be done by advanced AI?
In that case, substrate-needs convergence would not apply, or only apply to a limited extent.
There is still a concern about what those bio-engineered creatures, used in practice as slaves to automate our intellectual and physical work, would bring about over the long-term.
If there is a successful attempt by them to ‘upload’ their cognition onto networked machinery, then we’re stuck with the substrate-needs convergence problem again.