I am honestly very confused on how Forrest is so confident that radical positive changes will not happen in our lifetime.
More importantly, he seems to be complaining that his opponents have different goals, and claims they’re selectively rational. Heads up, but rational behavior can only be determined once what goals you have are determined. Now, to him, his goals probably are much less selfish than those that want AI progress to speed up, so it’s not rational for AI capabilities to increase. I too do not think AI progress is beneficial, and believe it probably is harmful, so I’d slowdown on the progress too.
This is critical, because Forrest is misidentifying why AI progress people want AI to progress. The fact that they have very different goals compared to you is the reason why they want AI to progress, and not a rationality failure.
Another critical crux is I am far more optimistic than Forrest or Remmelt on AGI Alignment working out in the end. If I had a pessimism level comparable to Forrest or Remmelt, I too would probably advocate far more around governance strategies.
This is for several reasons:
My general prior is most problems are solvable. This doesn’t always occur, see the halting problem’s unsolvability, or the likely non-solvability of a perpetual motion machine, but my prior is if there isn’t a theorem prohibiting it and it doesn’t rely on violating the laws of physics, I’d say it solvable. And AGI alignment is in this spot.
I believe alignment is progressing, not enough to be clear, but if AI alignment was as well resourced as AI capabilities research, then I’d give it a fair shot of solving the problem.
Finally, time. In the more conservative story described here, it still takes 20-30 years, and while AGI now would probably be incompatible with life due to instrumental convergence and inner alignment failures, so long as you have extremely pessimistic beliefs about progress in AI alignment, this is the type of time frame where I’d place 60% probability on having a working solution to the AGI alignment problem due to progress on it.
That prior for most problems being solvable is not justified. For starters, because you did not provide any reasons above to justify why beneficial AGI is not like a perpetual motion machine, AKA a “perpetual general benefit machine”.
Again no reasons given for the belief that AGI alignment is “progressing” or would have a “fair shot” of solving “the problem” if as well resourced as capabilities research. Basically nothing to argue against, because you are providing no arguments yet.
No reasons given, again. Presents instrumental convergence and intrinsic optimisation misalignment failures as the (only) threat models in terms of artificial general intelligence incompatibility with organic DNA-based life. Overlooks substrate-needs convergence.
Always happy to chat further about the substantive arguments. I was initially skeptical of Forrest’s “AGI-alignment is impossible” claim. But after probing and digging into this question intensely over the last year, I could not find anything unsound (in terms of premises) or invalid (in terms of logic) about his core arguments.
I am honestly very confused on how Forrest is so confident that radical positive changes will not happen in our lifetime.
More importantly, he seems to be complaining that his opponents have different goals, and claims they’re selectively rational. Heads up, but rational behavior can only be determined once what goals you have are determined. Now, to him, his goals probably are much less selfish than those that want AI progress to speed up, so it’s not rational for AI capabilities to increase. I too do not think AI progress is beneficial, and believe it probably is harmful, so I’d slowdown on the progress too.
This is critical, because Forrest is misidentifying why AI progress people want AI to progress. The fact that they have very different goals compared to you is the reason why they want AI to progress, and not a rationality failure.
Another critical crux is I am far more optimistic than Forrest or Remmelt on AGI Alignment working out in the end. If I had a pessimism level comparable to Forrest or Remmelt, I too would probably advocate far more around governance strategies.
This is for several reasons:
My general prior is most problems are solvable. This doesn’t always occur, see the halting problem’s unsolvability, or the likely non-solvability of a perpetual motion machine, but my prior is if there isn’t a theorem prohibiting it and it doesn’t rely on violating the laws of physics, I’d say it solvable. And AGI alignment is in this spot.
I believe alignment is progressing, not enough to be clear, but if AI alignment was as well resourced as AI capabilities research, then I’d give it a fair shot of solving the problem.
Finally, time. In the more conservative story described here, it still takes 20-30 years, and while AGI now would probably be incompatible with life due to instrumental convergence and inner alignment failures, so long as you have extremely pessimistic beliefs about progress in AI alignment, this is the type of time frame where I’d place 60% probability on having a working solution to the AGI alignment problem due to progress on it.
Responding below:
That prior for most problems being solvable is not justified. For starters, because you did not provide any reasons above to justify why beneficial AGI is not like a perpetual motion machine, AKA a “perpetual general benefit machine”.
See reasons to shift your prior: https://www.lesswrong.com/posts/Qp6oetspnGpSpRRs4/list-3-why-not-to-assume-on-prior-that-agi-alignment
Again no reasons given for the belief that AGI alignment is “progressing” or would have a “fair shot” of solving “the problem” if as well resourced as capabilities research. Basically nothing to argue against, because you are providing no arguments yet.
No reasons given, again. Presents instrumental convergence and intrinsic optimisation misalignment failures as the (only) threat models in terms of artificial general intelligence incompatibility with organic DNA-based life. Overlooks substrate-needs convergence.
I’ll concede here that I unfortunately do not have good arguments, and I’m updating towards pessimism regarding the alignment problem.
Appreciating your honesty, genuinely!
Always happy to chat further about the substantive arguments. I was initially skeptical of Forrest’s “AGI-alignment is impossible” claim. But after probing and digging into this question intensely over the last year, I could not find anything unsound (in terms of premises) or invalid (in terms of logic) about his core arguments.