Hypothesis: Maybe you’re actually not considering a reporter i that always use an intermediate model; but instead a reporter i’ that does translations on hard questions, and just uses the intermediate model on questions where it’s confident that the intermediate model understands everything relevant. I see three different possible issues with that idea:
1. To do this, i’ needs an efficient way (ie one that doesn’t scale with the size of the predictor) to (on at least some inputs) be highly confident that the intermediate model understands everything relevant about the situation. I think this is a reasonable “worst-case” assumption, but I’m not sure. If you’re using it, I’d be curious to know.
2. Even when the reporter gets inputs that the intermediate model fully understands, it seems like the reporter will run into issues if its sampling-process (for generating distributions) runs into some inputs that the intermediate model doesn’t fully understand. (I.e., if i’ gets an input for which X1 is true, and then simulates many more random inputs for which X1 is true, and by chance X2 is true for one of them, then the reporter will have to do translation on that sample). Which makes it seem unlikely that i’ wouldn’t have to do translation at least once per input.
3. If the model is confident that the intermediate model understands everything relevant about the situation, it seems more efficient to return the intermediate model’s answer to the question at hand than to loop over it many times, trying to fix correlations. So really we should get a reporter i″ that does translation on the predictor on hard questions and returns an intermediate model’s latent knowledge on easy questions. That seems like an ok reporter to get.
Hypothesis: Maybe you’re actually not considering a reporter i that always use an intermediate model; but instead a reporter i’ that does translations on hard questions, and just uses the intermediate model on questions where it’s confident that the intermediate model understands everything relevant. I see three different possible issues with that idea:
1. To do this, i’ needs an efficient way (ie one that doesn’t scale with the size of the predictor) to (on at least some inputs) be highly confident that the intermediate model understands everything relevant about the situation. I think this is a reasonable “worst-case” assumption, but I’m not sure. If you’re using it, I’d be curious to know.
2. Even when the reporter gets inputs that the intermediate model fully understands, it seems like the reporter will run into issues if its sampling-process (for generating distributions) runs into some inputs that the intermediate model doesn’t fully understand. (I.e., if i’ gets an input for which X1 is true, and then simulates many more random inputs for which X1 is true, and by chance X2 is true for one of them, then the reporter will have to do translation on that sample). Which makes it seem unlikely that i’ wouldn’t have to do translation at least once per input.
3. If the model is confident that the intermediate model understands everything relevant about the situation, it seems more efficient to return the intermediate model’s answer to the question at hand than to loop over it many times, trying to fix correlations. So really we should get a reporter i″ that does translation on the predictor on hard questions and returns an intermediate model’s latent knowledge on easy questions. That seems like an ok reporter to get.