I think incremental change is a bit overrated. Sure, if you have something that performs so well that chasing 1% improvements is worth it, then go for it. But don’t keep tweaking forever: you’ll get most of the gains in the first few months, and they will total about +20%, or maybe +50% if you’re a hero.
If your current thing doesn’t perform so well, it’s more cost-effective to look for big things that could bring +100% or +1000%. A/B tests are useful for that too, but need to be done differently:
Come up with a big thing that could have big impact. For example, shortform.
Identify the assumptions behind that thing. For example, “users will write shortform” or “users will engage with others’ shortform”.
Come up with cheap ways to test these assumptions. For example, “check the engagement on existing posts that are similar to shortform” or “suggest to some power users that they should make shortform posts and see how much engagement they get”. At this step you may end up looking at metrics, looking at competitors, or running cheap A/B tests.
Based on the previous steps, change your mind about which thing you want to build, and repeat these steps until you’re pretty sure it will succeed.
This line of thinking makes a major assumption which has, in my experience, been completely wrong: the assumption that a “big thing” in terms of impact is also a “big thing” in terms of engineering effort. I have seen many changes which are only small tweaks from an engineering standpoint, but produce 25% or 50% increase in a metric all on their own—things like making a button bigger, clarifying/shortening some text, changing something from red to green, etc. Design matters, it’s relatively easy to change, but we don’t know how to change it usefully without tests.
Agreed—I’ve seen, and made, quite a few such changes as well. After each big upheaval it’s worth spending some time grabbing the low hanging fruit. My only gripe is that I don’t think this type of change is sufficient over a project’s lifetime. Deeper product change has a way of becoming necessary.
I think incremental change is a bit overrated. Sure, if you have something that performs so well that chasing 1% improvements is worth it, then go for it. But don’t keep tweaking forever: you’ll get most of the gains in the first few months, and they will total about +20%, or maybe +50% if you’re a hero.
If your current thing doesn’t perform so well, it’s more cost-effective to look for big things that could bring +100% or +1000%. A/B tests are useful for that too, but need to be done differently:
Come up with a big thing that could have big impact. For example, shortform.
Identify the assumptions behind that thing. For example, “users will write shortform” or “users will engage with others’ shortform”.
Come up with cheap ways to test these assumptions. For example, “check the engagement on existing posts that are similar to shortform” or “suggest to some power users that they should make shortform posts and see how much engagement they get”. At this step you may end up looking at metrics, looking at competitors, or running cheap A/B tests.
Based on the previous steps, change your mind about which thing you want to build, and repeat these steps until you’re pretty sure it will succeed.
Build the thing.
This is roughly the procedure we usually follow.
This line of thinking makes a major assumption which has, in my experience, been completely wrong: the assumption that a “big thing” in terms of impact is also a “big thing” in terms of engineering effort. I have seen many changes which are only small tweaks from an engineering standpoint, but produce 25% or 50% increase in a metric all on their own—things like making a button bigger, clarifying/shortening some text, changing something from red to green, etc. Design matters, it’s relatively easy to change, but we don’t know how to change it usefully without tests.
Agreed—I’ve seen, and made, quite a few such changes as well. After each big upheaval it’s worth spending some time grabbing the low hanging fruit. My only gripe is that I don’t think this type of change is sufficient over a project’s lifetime. Deeper product change has a way of becoming necessary.