… running these [A/B] tests allows them to make products that are better than if they didn’t run the tests.
Empirically, as a trend across the industry, this has turned out to be false. “Design by A/B test” has dramatically eroded the quality of UI/UX design over the last 10-15 years.
Giving everyone worse outcomes to make sure everyone always gets identical outcomes would not be an improvement.
On the contrary, it quite often would be an improvement—and a big one. Not only are “worse” outcomes by the metrics usually used in A/B tests often not even actually worse by any measure that users might care about, but the gains from consistency (both synchronic and diachronic) are commonly underestimated (for example—clearly—by you); in fact such gains are massive, and compound in the long run. Inconsistency, on the other hand, has many detrimental knock-on effects (increased centralization and dependence on unaccountable authorities, un-democratization of expertise, increased education and support costs, the creation and maintenance of a self-perpetuating expert class and the power imbalances that result—all of these things are either directly caused, or exacerbated, by the synchronic and diachronic UI inconsistency that is rampant in today’s software).
Addendum to my other comment:
Empirically, as a trend across the industry, this has turned out to be false. “Design by A/B test” has dramatically eroded the quality of UI/UX design over the last 10-15 years.
On the contrary, it quite often would be an improvement—and a big one. Not only are “worse” outcomes by the metrics usually used in A/B tests often not even actually worse by any measure that users might care about, but the gains from consistency (both synchronic and diachronic) are commonly underestimated (for example—clearly—by you); in fact such gains are massive, and compound in the long run. Inconsistency, on the other hand, has many detrimental knock-on effects (increased centralization and dependence on unaccountable authorities, un-democratization of expertise, increased education and support costs, the creation and maintenance of a self-perpetuating expert class and the power imbalances that result—all of these things are either directly caused, or exacerbated, by the synchronic and diachronic UI inconsistency that is rampant in today’s software).