The common way to evaluate model accuracy in machine learning contexts is that you have a bunch of samples of the “ground truth” that is to be predicted; e.g. classified images for supervised learning. And then you evaluate the model on those samples. That is the sort of accuracy measure I had in mind when writing the post, because that is what gets used in practice.
That’s what gets used for supervised or unsupervised learning, but your post started out with “Suppose we want to create an agent AI”, and there’s no straightforward way of interpreting systems trained with those techniques as agents. Perhaps you intended for some such system to be used as the “model” subsystem of an agent AI, but in that case I think the problem really is basically what I said: the actor should be defining what information it wants to get out of the model, and the model should be optimized to supply that information, and if it isn’t, that model won’t do as well at providing the information the actor needs.
I don’t think “amount of information contained” even sounds like a property of a model that anyone would think they should care about, absent some detail about what that information is about. Otherwise a model that knows nothing but a sufficiently massive number of digits of pi would be better than one that can answer any question you have about the real world but knows pi to only 50 decimal places. “Percent of questions in the test set answered correctly” does sound possibly useful, if you want to get answers to questions drawn from the same distribution. “Percent of questions I actually ask, weighted by how much value I get from having that particular question answered correctly” would be an even better metric (with the defect of being impossible to directly optimize for), of course, but the long book about who lives where and the library describing the death chamber don’t even seem to live up to the minimal “this answers the kind of questions I want to ask” criterion.
I mean you can consider something like Dreamer, an RL agent I’ve seen. It trains a model to predict the dynamics of a system, and then trains the behavior using that model. I don’t see how this RL agent is compatible with your comment.
That’s what gets used for supervised or unsupervised learning, but your post started out with “Suppose we want to create an agent AI”, and there’s no straightforward way of interpreting systems trained with those techniques as agents. Perhaps you intended for some such system to be used as the “model” subsystem of an agent AI, but in that case I think the problem really is basically what I said: the actor should be defining what information it wants to get out of the model, and the model should be optimized to supply that information, and if it isn’t, that model won’t do as well at providing the information the actor needs.
I don’t think “amount of information contained” even sounds like a property of a model that anyone would think they should care about, absent some detail about what that information is about. Otherwise a model that knows nothing but a sufficiently massive number of digits of pi would be better than one that can answer any question you have about the real world but knows pi to only 50 decimal places. “Percent of questions in the test set answered correctly” does sound possibly useful, if you want to get answers to questions drawn from the same distribution. “Percent of questions I actually ask, weighted by how much value I get from having that particular question answered correctly” would be an even better metric (with the defect of being impossible to directly optimize for), of course, but the long book about who lives where and the library describing the death chamber don’t even seem to live up to the minimal “this answers the kind of questions I want to ask” criterion.
I mean you can consider something like Dreamer, an RL agent I’ve seen. It trains a model to predict the dynamics of a system, and then trains the behavior using that model. I don’t see how this RL agent is compatible with your comment.