Without getting into the details of the paper, this seems to be contradicted by evidence from people.
Humans are clearly generally intelligent, and out of anyone else’s control.
Human variability is obviously only a tiny fraction of the possible range of intelligences, given how closely related we all are both genetically and environmentally. Yet human goals have huge ranges, including plenty that include X-risk.
E.g.
Kill all inferior races (Hitler)
Solve Fermat’s last theorem (Andrew Wiles)
Enter Nirvana (A budhist)
Win the lottery at any cost (Danyal Hussein)
So I would be deeply suspicious of any claim that a general intelligence would be limited in the range of its potential goals. I would similarly be deeply suspicious of any claim that the goals wouldn’t be stable, or at least stable enough to carry out—Andrew Wiles, Hitler, and the Budhist are all capable of carrying out their very different goals over long time periods. I would want to understand why the argument doesn’t apply to them, but does to artificial intelligences.
Without getting into the details of the paper, this seems to be contradicted by evidence from people.
Humans are clearly generally intelligent, and out of anyone else’s control.
Human variability is obviously only a tiny fraction of the possible range of intelligences, given how closely related we all are both genetically and environmentally. Yet human goals have huge ranges, including plenty that include X-risk.
E.g.
Kill all inferior races (Hitler)
Solve Fermat’s last theorem (Andrew Wiles)
Enter Nirvana (A budhist)
Win the lottery at any cost (Danyal Hussein)
So I would be deeply suspicious of any claim that a general intelligence would be limited in the range of its potential goals. I would similarly be deeply suspicious of any claim that the goals wouldn’t be stable, or at least stable enough to carry out—Andrew Wiles, Hitler, and the Budhist are all capable of carrying out their very different goals over long time periods. I would want to understand why the argument doesn’t apply to them, but does to artificial intelligences.