I actually agree with the overall judgement there—optimizing simple metrics really hard is mainly useful for things like e.g. landing pages, where the goals really are pretty simple and there’s not too much danger of Goodharting. Lesswrong mostly isn’t like that, and most of the value in micro-optimizing would be in the knowledge gained, rather than the concrete result of increasing a metric. I do think there’s a lot of knowledge there to gain, and I think our design-level decisions are currently far away from the pareto frontier in ways that won’t be obvious until the micro-optimization loop starts up.
I will also say that the majority of people I’ve worked with have dramatically underestimated the magnitude of impact this sort of thing has until they saw it happen first-hand, for whatever that’s worth. (I first saw it in action at a company which achieved supercritical virality for a short time, and A/B-test-driven micro-optimization was the main tool responsible for that.) If this were a start-up, and we needed strong new user and engagement metrics to get our next round of funding, then I’d say it should be the highest priority. But this isn’t a startup, and I totally agree that A/B tests won’t solve the most crucial uncertainties.
I actually agree with the overall judgement there—optimizing simple metrics really hard is mainly useful for things like e.g. landing pages, where the goals really are pretty simple and there’s not too much danger of Goodharting. Lesswrong mostly isn’t like that, and most of the value in micro-optimizing would be in the knowledge gained, rather than the concrete result of increasing a metric. I do think there’s a lot of knowledge there to gain, and I think our design-level decisions are currently far away from the pareto frontier in ways that won’t be obvious until the micro-optimization loop starts up.
I will also say that the majority of people I’ve worked with have dramatically underestimated the magnitude of impact this sort of thing has until they saw it happen first-hand, for whatever that’s worth. (I first saw it in action at a company which achieved supercritical virality for a short time, and A/B-test-driven micro-optimization was the main tool responsible for that.) If this were a start-up, and we needed strong new user and engagement metrics to get our next round of funding, then I’d say it should be the highest priority. But this isn’t a startup, and I totally agree that A/B tests won’t solve the most crucial uncertainties.