The situation seems to me comparable to one where we upload Hugh and then let him do what he wants, such as optimizing himself by replacing parts of himself by machine learning predictors with correlating results. The hope here then sounds like that this is fine so long as we perform differential testing to limit accidental drift.
How does this differ from just running Hugh?
Hugh is some human, Arthur is a cheap AI. For the obvious example today, compare:
Get mechanical turkers to label training images. Train an AI to predict the label they would assign. Use that AI to label images.
Use mechanical turkers to label images.
The second one is orders of magnitude more expensive and higher latency.
The situation seems to me comparable to one where we upload Hugh and then let him do what he wants, such as optimizing himself by replacing parts of himself by machine learning predictors with correlating results. The hope here then sounds like that this is fine so long as we perform differential testing to limit accidental drift.