Luckily I don’t need to show that sufficiently smart AIs don’t engage in trial and error. All I need to show is that they almost certainly do not engage in the particular kind of trial of running a computer program without already knowing whether the program is satisfactory.
you have thereby defined sufficiently smart as AIs that satisfy this requirement. this is not the case. many likely designs for AIs well above human level will have need to actually run parts of programs to get their results. perhaps usually fairly small ones.
trial and error is sometimes needed internal to learning, there are always holes in knowledge
Luckily I don’t need to show that sufficiently smart AIs don’t engage in trial and error. All I need to show is that they almost certainly do not engage in the particular kind of trial of running a computer program without already knowing whether the program is satisfactory.
you have thereby defined sufficiently smart as AIs that satisfy this requirement. this is not the case. many likely designs for AIs well above human level will have need to actually run parts of programs to get their results. perhaps usually fairly small ones.