This comment is a poorly-organized brain dump which serves as a convenient gathering place for what I’ve learned after several days of arguing with every MIRI critic I could find. It will probably get it’s own expanded post in the future, and if I have the time I may try to build a near-comprehensive list.
I’ve come to understand that criticisms of MIRI’s version of the intelligence explosion hypothesis and the penumbra of ideas around it fall into two permeable categories:
Those that criticize MIRI as an organization or the whole FAI enterprise (people making these arguments may or may not be concerned about the actual IE) and those that attack object-level claims made by MIRI.
Broad Criticisms
1a) Why worry about this now, instead of in the distant future, given the abysmal performance of attempts to predict AI?
1b) Why take MIRI seriously when there are so many expert opinions that diverge?
1c) Aren’t MIRI and LW just an Eliezer-worshipping cult?
1d) Is it even possible to do this kind of theoretical work so far in advance of actual testing and experimentation?
1e) The whole argument can be dismissed as it pattern matches other doomsday scenarios, almost all of which have been bullshit.
Specific Criticisms
2a) General intelligence is what we’re worried about here, and it may prove much harder to build than we’re anticipating.
2g) A self-improvement cascade will likely hit a wall at sub-superintelligent levels.
2h) Divergence Issue: all functioning AI systems have built-in sanity checks which take short-form goal statements and unpack them in ways that take account of constraints and context (???). It is actually impossible to build an AI which does not do this (???), and thus there can be no runaway SAI which is given a simple short-form goal and then carries it to ridiculous logical extremes (I WOULD BE PARTICULARLY INTERESTED IN SOMEONE ADDRESSING THIS).
This comment is a poorly-organized brain dump which serves as a convenient gathering place for what I’ve learned after several days of arguing with every MIRI critic I could find. It will probably get it’s own expanded post in the future, and if I have the time I may try to build a near-comprehensive list.
I’ve come to understand that criticisms of MIRI’s version of the intelligence explosion hypothesis and the penumbra of ideas around it fall into two permeable categories:
Those that criticize MIRI as an organization or the whole FAI enterprise (people making these arguments may or may not be concerned about the actual IE) and those that attack object-level claims made by MIRI.
Broad Criticisms
1a) Why worry about this now, instead of in the distant future, given the abysmal performance of attempts to predict AI?
1b) Why take MIRI seriously when there are so many expert opinions that diverge?
1c) Aren’t MIRI and LW just an Eliezer-worshipping cult?
1d) Is it even possible to do this kind of theoretical work so far in advance of actual testing and experimentation?
1e) The whole argument can be dismissed as it pattern matches other doomsday scenarios, almost all of which have been bullshit.
Specific Criticisms
2a) General intelligence is what we’re worried about here, and it may prove much harder to build than we’re anticipating.
2b) Tool AIs won’t be as dangerous as agent AIs.
2c) Why not just build an Oracle?
2d) the FOOM will be distributed and slow, not fast and localized.
2e) Dumb Superintelligence, i.e. nothing worth of the name could possibly misinterpret a goal like ‘make humans happy’
2f) Even FAI isn’t a guarantee
2g) A self-improvement cascade will likely hit a wall at sub-superintelligent levels.
2h) Divergence Issue: all functioning AI systems have built-in sanity checks which take short-form goal statements and unpack them in ways that take account of constraints and context (???). It is actually impossible to build an AI which does not do this (???), and thus there can be no runaway SAI which is given a simple short-form goal and then carries it to ridiculous logical extremes (I WOULD BE PARTICULARLY INTERESTED IN SOMEONE ADDRESSING THIS).