a naive estimate of the expected test loss would be the average training loss using samples of the posterior.
That’s exactly the problem—that is generally not a good estimate of the expected test loss. It isn’t even an unbiased estimate. It’s just completely wrong.
The right way to do this is to just calculate the expected test loss.
That’s exactly the problem—that is generally not a good estimate of the expected test loss. It isn’t even an unbiased estimate. It’s just completely wrong.
The right way to do this is to just calculate the expected test loss.