Trying to optimize a metric without even having a test framework in place adds a lot of evidence to that story—certainly in my own start-up experience, we never had any idea what we were doing until well after the test framework was in place (at any of the companies I’ve worked at). Analytics more generally were also always crucial for figuring out where the low-hanging fruit was and which projects to prioritize, and it sounds like you guys are currently still flying blind in that department.
I think I agree with the general spirit here. Throughout my year with the LessWrong team, I’ve been progressively building out analytics infrastructure to reduce my sense of the “flying blind” you speak of. We’re not done yet, but I’ve now got a lot of data at my fingertips. I think the disagreement here would be over whether anything short of A/B testing is valuable. I’m pretty sure that it is.
I think I agree with the general spirit here. Throughout my year with the LessWrong team, I’ve been progressively building out analytics infrastructure to reduce my sense of the “flying blind” you speak of. We’re not done yet, but I’ve now got a lot of data at my fingertips. I think the disagreement here would be over whether anything short of A/B testing is valuable. I’m pretty sure that it is.