I want to discuss the specific example you picked: Etsy & A/B (or generally, data-driven) testing.
I’ll start by agreeing to your premise in the abstract: Etsy could probably have done better by using a better A/B testing methodology and doing so sooner.
But: while superior tools used effectively are superior, they also tend to be harder to use and to cause more damage when misused. I’ve seen a bunch of math-based data-driven analysis that wasn’t worth a damn because they were misused. The more sophisticated these tools become, the easier they are to misuse.
A popular cautionary tale in that space is “Steve Jobs vs focus groups”. Focus groups are a primitive but, I’d argue, *scientific* method to determine what your customers want. Yet what it comes up with is often lackluster to the point where a visionary with a intuition and taste can run circles around them. Sure, in this primitive case we can easily point to some design flaws in the process. For instance, bringing various people together to discuss a design is likely to produce a compromise design that truly satisfies no one. But is A/B testing free of any such psychological flaws? I think not, and now you also risk screwing up the statistical analysis in one hundred different ways.
Second, trade-offs. If Etsy chose to perfect it’s A/B testing methodology, it has to forego doing other things, because it does not have illimited resources (even if it had, if an entity becomes too large to coordinate effectively that comes with a slew of issues — this is well documented (the mythical man month, etc)). Could it be that their unsophisticated/late application of A/B was an effective trade-off in terms of resource use (à la 20⁄80 principle)?
I think a part of instrumental rationality is the ability to decide which tools to deploy.
Generally, more sophisticated tools can yield better results, but come with an inherent cost. You seem to ignore this basic reality here. Cases where it’s as easy to do the smart thing as to do the dumb thing exist, but they’re not everywhere—especially in business context (and even more so in startups), where there is a strong evolutionary pressure to adopt these low-hanging fruits.
I want to discuss the specific example you picked: Etsy & A/B (or generally, data-driven) testing.
I’ll start by agreeing to your premise in the abstract: Etsy could probably have done better by using a better A/B testing methodology and doing so sooner.
But: while superior tools used effectively are superior, they also tend to be harder to use and to cause more damage when misused. I’ve seen a bunch of math-based data-driven analysis that wasn’t worth a damn because they were misused. The more sophisticated these tools become, the easier they are to misuse.
A popular cautionary tale in that space is “Steve Jobs vs focus groups”. Focus groups are a primitive but, I’d argue, *scientific* method to determine what your customers want. Yet what it comes up with is often lackluster to the point where a visionary with a intuition and taste can run circles around them. Sure, in this primitive case we can easily point to some design flaws in the process. For instance, bringing various people together to discuss a design is likely to produce a compromise design that truly satisfies no one. But is A/B testing free of any such psychological flaws? I think not, and now you also risk screwing up the statistical analysis in one hundred different ways.
Second, trade-offs. If Etsy chose to perfect it’s A/B testing methodology, it has to forego doing other things, because it does not have illimited resources (even if it had, if an entity becomes too large to coordinate effectively that comes with a slew of issues — this is well documented (the mythical man month, etc)). Could it be that their unsophisticated/late application of A/B was an effective trade-off in terms of resource use (à la 20⁄80 principle)?
I think a part of instrumental rationality is the ability to decide which tools to deploy.
Generally, more sophisticated tools can yield better results, but come with an inherent cost. You seem to ignore this basic reality here. Cases where it’s as easy to do the smart thing as to do the dumb thing exist, but they’re not everywhere—especially in business context (and even more so in startups), where there is a strong evolutionary pressure to adopt these low-hanging fruits.