These all seem like reasonable heuristics for a first-pass judgement of a field, but once you’ve actually engaged with the arguments you should either be finding actual real disagreements about assumptions/argument-validity or not, and if not you should update a bunch to ‘it looks like they may be right’. You can’t expect everyone to actually engage with your field, but I will certainly ask people to, and also continue to get annoyed when the above arguments are given as slam-dunks which people think can substitute for actually engaging with advocates.
I think that sounds about right. Collecting the arguments in one place is definitely helpful, and I think they carry some weight as initial heuristics, which this post helps clarify.
But I also think the technical arguments should (mostly) screen off the heuristics; the heuristics are better for evaluating whether it’s worth paying attention to the details. By the time you’re having a long debate, it’s better to spend (at least some) time looking instead of continuing to rely on the heuristics. Rhymes with Argument Screens Off Authority. (And in both cases, only mostly screens off.)
The Marxist arguments for the collapse of capitalism always sounded handwavey to me, but perhaps you could link me to something that would have sounded persuasive in the past?
After thinking about past theories that were falsified, I think that the heuristic is still strong enough to make us pretty uncertain about AI x-risk. In Yudkowsky’s model of AI progress, I think AI x-risk would be something like 99%, but taking into account that theories can be wrong in unexpected ways I’d guess it is more like 60% (20% that Yudkowsky’s model is right, 40% that it is wrong but AI x-risk happens for a different reason).
Of course even with 60% risk, AI alignment is extremely important.
I haven’t thought too hard about the “past theory” accuracy though. That’s part of why I made this and the previous post; I’m trying to figure it out.
I think the right way of thinking about that aspect is more: there are a bunch of methodologies to analyze the AI x-risk situation, and only one of the methods seems to give tremendously high credence to FOOM & DOOM.
Not so much a ‘you could be wrong’ argument, because I do think that in the Eliezer framework, it brings little comfort if you’re wrong about your picture of how intelligence works, since its highly improbable you’re wrong about something that makes a problem easier rather than harder.
This leads to the natural next question: what alternative methodologies & why do you have the faith you do in them when contrasted with the set of methodologies claiming FOOM & DOOM. And possibly a discussion of whether or not those alternative methodologies actually support the anti-FOOM & DOOM position. For example, you may claim that extrapolating lines on graphs says humans will continue to flourish, but the actual graphs we have are about GDP, and other crude metrics of human welfare. Those could very well continue without the human flourishing part, and indeed if they do continue indefinitely we should expect human welfare to be sacrificed to the gods of straight-lines-on-graphs to achieve this outcome.
These all seem like reasonable heuristics for a first-pass judgement of a field, but once you’ve actually engaged with the arguments you should either be finding actual real disagreements about assumptions/argument-validity or not, and if not you should update a bunch to ‘it looks like they may be right’. You can’t expect everyone to actually engage with your field, but I will certainly ask people to, and also continue to get annoyed when the above arguments are given as slam-dunks which people think can substitute for actually engaging with advocates.
Good on you for collecting them here.
I think that sounds about right. Collecting the arguments in one place is definitely helpful, and I think they carry some weight as initial heuristics, which this post helps clarify.
But I also think the technical arguments should (mostly) screen off the heuristics; the heuristics are better for evaluating whether it’s worth paying attention to the details. By the time you’re having a long debate, it’s better to spend (at least some) time looking instead of continuing to rely on the heuristics. Rhymes with Argument Screens Off Authority. (And in both cases, only mostly screens off.)
I think you’re overestimating the strength of the arguments and underestimating the strength of the heuristic.
All the Marxist arguments for why capitalism would collapse were probably very strong and intuitive, but they lost to the law of straight lines.
I think you have to imagine yourself in that position and think about how you would feel and think about the problem.
The Marxist arguments for the collapse of capitalism always sounded handwavey to me, but perhaps you could link me to something that would have sounded persuasive in the past?
One side of the debate has a duty to engage with the actual arguments; the other side has a duty to provide them.
https://twitter.com/ESYudkowsky/status/1647396683423305728
After thinking about past theories that were falsified, I think that the heuristic is still strong enough to make us pretty uncertain about AI x-risk. In Yudkowsky’s model of AI progress, I think AI x-risk would be something like 99%, but taking into account that theories can be wrong in unexpected ways I’d guess it is more like 60% (20% that Yudkowsky’s model is right, 40% that it is wrong but AI x-risk happens for a different reason).
Of course even with 60% risk, AI alignment is extremely important.
I haven’t thought too hard about the “past theory” accuracy though. That’s part of why I made this and the previous post; I’m trying to figure it out.
I think the right way of thinking about that aspect is more: there are a bunch of methodologies to analyze the AI x-risk situation, and only one of the methods seems to give tremendously high credence to FOOM & DOOM.
Not so much a ‘you could be wrong’ argument, because I do think that in the Eliezer framework, it brings little comfort if you’re wrong about your picture of how intelligence works, since its highly improbable you’re wrong about something that makes a problem easier rather than harder.
This leads to the natural next question: what alternative methodologies & why do you have the faith you do in them when contrasted with the set of methodologies claiming FOOM & DOOM. And possibly a discussion of whether or not those alternative methodologies actually support the anti-FOOM & DOOM position. For example, you may claim that extrapolating lines on graphs says humans will continue to flourish, but the actual graphs we have are about GDP, and other crude metrics of human welfare. Those could very well continue without the human flourishing part, and indeed if they do continue indefinitely we should expect human welfare to be sacrificed to the gods of straight-lines-on-graphs to achieve this outcome.