Ah, I think I see, thanks for explaining. So even when you talk about amplifying f, you mean a certain way of extending human predictions to more complicated background information (e.g. via breaking down Z into chunks and then using copies of f that have been trained on smaller Z), not fine-tuning f to make better predictions. Or maybe some amount of fine-tuning for “better” predictions by some method of eliciting its own standards, but not by actually comparing it to the ground truth.
This (along with eventually reading your companion post) also helps resolve the confusion I was having over what exactly was the prior in “learning the prior”—Z is just like a latent space, and f is the decoder from Z to predictions. My impression is that your hope is that if Z and f start out human-like, then this is like specifying the “programming language” of a universal prior, so that search for highly-predictive Z, decoded through f, will give something that uses human concepts in predicting the world.
So even when you talk about amplifying f, you mean a certain way of extending human predictions to more complicated background information (e.g. via breaking down Z into chunks and then using copies of f that have been trained on smaller Z), not fine-tuning f to make better predictions.
That’s right, f is either imitating a human, or it’s trained by iterated amplification / debate—in any case the loss function is defined by the human. In no case is f optimized to make good predictions about the underlying data.
My impression is that your hope is that if Z and f start out human-like, then this is like specifying the “programming language” of a universal prior, so that search for highly-predictive Z, decoded through f, will give something that uses human concepts in predicting the world.
Z should always be a human-readable (or amplified-human-readable) latent; it will necessarily remain human-readable because it has no purpose other than to help a human make predictions. f is going to remain human-like because it’s predicting what the human would say (or what the human-consulting-f would say etc.).
The amplified human is like the programming language of the universal prior, Z is like the program that is chosen (or slightly more precisely: Z is like a distribution over programs, described in a human-comprehensible way) and f is an efficient distillation of the intractable ideal.
Ah, I think I see, thanks for explaining. So even when you talk about amplifying f, you mean a certain way of extending human predictions to more complicated background information (e.g. via breaking down Z into chunks and then using copies of f that have been trained on smaller Z), not fine-tuning f to make better predictions. Or maybe some amount of fine-tuning for “better” predictions by some method of eliciting its own standards, but not by actually comparing it to the ground truth.
This (along with eventually reading your companion post) also helps resolve the confusion I was having over what exactly was the prior in “learning the prior”—Z is just like a latent space, and f is the decoder from Z to predictions. My impression is that your hope is that if Z and f start out human-like, then this is like specifying the “programming language” of a universal prior, so that search for highly-predictive Z, decoded through f, will give something that uses human concepts in predicting the world.
Is that somewhat in the right ballpark?
That’s right, f is either imitating a human, or it’s trained by iterated amplification / debate—in any case the loss function is defined by the human. In no case is f optimized to make good predictions about the underlying data.
Z should always be a human-readable (or amplified-human-readable) latent; it will necessarily remain human-readable because it has no purpose other than to help a human make predictions. f is going to remain human-like because it’s predicting what the human would say (or what the human-consulting-f would say etc.).
The amplified human is like the programming language of the universal prior, Z is like the program that is chosen (or slightly more precisely: Z is like a distribution over programs, described in a human-comprehensible way) and f is an efficient distillation of the intractable ideal.