Unrelated to the effects of the outputs, I think the BFO approach might have the following limitation (this is inspired by your comment on a setup I once described):
It might be the case that in turn t′, when the programmers try to use the BFO to get answers to some hard questions, the “training distribution” at that point would be “easier” (or otherwise different) than the “test distribution” in some important sense, such that the BFO won’t provide useful answers. For example, perhaps a BFO couldn’t be used, soon after development, to answer: “If we give $10,000,000 to a research lab X for their project Y, will the first FDA-approved cure for Alzheimer’s be approved at least twice sooner?”.
But even if that’s the case, a BFO might be sufficiently useful, even if it could always answer only questions that are not very different than the previous ones. A gradual process of answering harder and harder questions might provide sufficient value sufficiently quickly.
Unrelated to the effects of the outputs, I think the BFO approach might have the following limitation (this is inspired by your comment on a setup I once described):
It might be the case that in turn t′, when the programmers try to use the BFO to get answers to some hard questions, the “training distribution” at that point would be “easier” (or otherwise different) than the “test distribution” in some important sense, such that the BFO won’t provide useful answers. For example, perhaps a BFO couldn’t be used, soon after development, to answer: “If we give $10,000,000 to a research lab X for their project Y, will the first FDA-approved cure for Alzheimer’s be approved at least twice sooner?”.
But even if that’s the case, a BFO might be sufficiently useful, even if it could always answer only questions that are not very different than the previous ones. A gradual process of answering harder and harder questions might provide sufficient value sufficiently quickly.