I think this is a very natural frame, but here is an alternative. Humans largely do not have large scale or long term goals/values relative to which they can be said to be making mistakes, that AI will help prevent in the future. Instead they care almost exclusively about local and short term concerns like satisfying physical desires and winning status games, and the fate of our universe depends largely on humanity’s side effects as they go about acting on these parochial interests. (Consider e.g. Warren Buffett’s interest in accumulating wealth and disinterest in how it’s actually spent, suggesting that he only cares about it as a status marker.)
This state of affairs may well persist into the AGI era, with status games taking on even more importance as physical desires are largely satisfied/satiated. (This assumes we avoid a bunch of even worse failure modes.)
Since status games are competitive games with necessarily winners and losers, in this scenario it doesn’t seem to make sense to say that humans will stop making mistakes with AI assistance.
I think this is a very natural frame, but here is an alternative. Humans largely do not have large scale or long term goals/values relative to which they can be said to be making mistakes, that AI will help prevent in the future. Instead they care almost exclusively about local and short term concerns like satisfying physical desires and winning status games, and the fate of our universe depends largely on humanity’s side effects as they go about acting on these parochial interests. (Consider e.g. Warren Buffett’s interest in accumulating wealth and disinterest in how it’s actually spent, suggesting that he only cares about it as a status marker.)
This state of affairs may well persist into the AGI era, with status games taking on even more importance as physical desires are largely satisfied/satiated. (This assumes we avoid a bunch of even worse failure modes.)
Since status games are competitive games with necessarily winners and losers, in this scenario it doesn’t seem to make sense to say that humans will stop making mistakes with AI assistance.