we can try to train a purely predictive model with only a world model but no optimization procedure or objective.
How might a “purely predictive model with only a world model but no optimization procedure” look like, when considering complicated domains and arbitrarily high predictive accuracy?
It seems plausible that a sufficiently accurate predictive model would use powerful optimization processes. For example, consider a predictive model that predicts the change in Apple’s stock price at some moment t (based on data until t). A sufficiently powerful model might, for example, search for solutions to some technical problem related to the development of the next iPhone (that is being revealed that day) in order to calculate the probability that Apple’s engineers overcame it.
If the model that is used as a Microscope AI does not use any optimization (search), how will it compute the probability that, say, Apple’s engineers will overcome a certain technical challenge?
That’s a good question. Perhaps it does make use of optimization but the model still has an overall passive relationship to the world compared to an active mesa-optimizer AI. I’m thinking about the difference between say, GPT-3 and the classic paperclip maximizer or other tiling AI.
This is just my medium-confidence understanding and may be different from what Evan Hubinger meant in that quote.
How might a “purely predictive model with only a world model but no optimization procedure” look like, when considering complicated domains and arbitrarily high predictive accuracy?
It seems plausible that a sufficiently accurate predictive model would use powerful optimization processes. For example, consider a predictive model that predicts the change in Apple’s stock price at some moment t (based on data until t). A sufficiently powerful model might, for example, search for solutions to some technical problem related to the development of the next iPhone (that is being revealed that day) in order to calculate the probability that Apple’s engineers overcame it.
I believe it would look like Microscope AI.
If the model that is used as a Microscope AI does not use any optimization (search), how will it compute the probability that, say, Apple’s engineers will overcome a certain technical challenge?
That’s a good question. Perhaps it does make use of optimization but the model still has an overall passive relationship to the world compared to an active mesa-optimizer AI. I’m thinking about the difference between say, GPT-3 and the classic paperclip maximizer or other tiling AI.
This is just my medium-confidence understanding and may be different from what Evan Hubinger meant in that quote.