My understanding of A/B testing is that you don’t need an explicit causal model , or a “big theory” in order to successfully use it, you mostly would be using intuitions gained from experience in order to test hypotheses like “users like the red page better than the blue page”, which has no explicit causal information.
Here you argue that intuitions gained from experience count as hypotheses just as much as causal theories do, and not only that, but that they tend to succeed more often than the big theories do. That depends on what you consider to be “success” I think. I agree that empirically gained intuitions probably have a lower failure rate than causal theories (you won’t do much worse than average) but what Eliezer is mainly arguing is that you won’t do much better than average, either.
And as far as you don’t mind just doing ok on average, that might be fine, then. But the main thing this book is grappling with is “how do I know when I can do a lot better than average?” And that seems to be dependent on whether or not you have a good “big theory” available.
My understanding of A/B testing is that you don’t need an explicit causal model , or a “big theory” in order to successfully use it, you mostly would be using intuitions gained from experience in order to test hypotheses like “users like the red page better than the blue page”, which has no explicit causal information.
Here you argue that intuitions gained from experience count as hypotheses just as much as causal theories do, and not only that, but that they tend to succeed more often than the big theories do. That depends on what you consider to be “success” I think. I agree that empirically gained intuitions probably have a lower failure rate than causal theories (you won’t do much worse than average) but what Eliezer is mainly arguing is that you won’t do much better than average, either.
And as far as you don’t mind just doing ok on average, that might be fine, then. But the main thing this book is grappling with is “how do I know when I can do a lot better than average?” And that seems to be dependent on whether or not you have a good “big theory” available.