Regarding the way Etsy was prioritizing development, it sounds like even their late stage “idea > validate > prototype” cycle is wrong. How can you “validate” before you have a prototype to get feedback on?
You need to be a little creative. I’ll give a trivial example that will probably be more enlightening if you went through the proofs C.S. bachelor students have to do of evaluating upper bounds of algorithms.
Lets say you hear about how matrix multiplication is a big problem in ML, and think you have a way to build an algorithm that should give you O(n*sqrt(n)) performance. Your idea happens to be an unusually complicated algorithm and thus a involves a complicated product; you think it will take you around 50 +-20 hours to prototype, and you don’t expect to achieve O(n*sqrt(n)) your first try. You could start your project now and build a prototype as quickly as possible, so that you can get a concrete impression of your algorithms viability. Or, you could first sit and consider for a couple hours and see if there are any ways to test or debunk the feasibility of your algorithm. If you choose the first option, you might get twenty hours in and realize your mistake only then. If you choose the second option, you might realize, just by pondering constraints for an hour, that matrix multiplication requires at least O(n^2) reads, and that your algorithm can’t in any universe work as fast as you expected it to, so no need to build a prototype without further considering where you’re going wrong.
Some ideas aren’t very easily validated. Art (I assume though I am not an artist) is an area where you pretty much just have a difficult to convey imagined finished product, and the best way to start validating is to build it and see if people like it. Lots of seed stage startups are founded to build things no one wants. The reason the team got funding anyways is often because the utility of the product they’re trying to create is difficult to “debunk” from the outset, or fails in hard-to-anticipate ways that aren’t caught by standard sanity checking.
However, outside of a few of those specific cases, most good ideas can be thought about. It might be impossible to completely rule out or in your hunch without building it first. But just by thinking about a proposed solution for the first twelve hours, you can usually establish some sort of probabilistic upper bound on the expected value and lower bound on the cost of implementation/maintenance. For a project that you think is going to take several months, even if you think there’s only a small chance you will figure out a flaw in its viability without prototyping, it’s worth it to sit down and do that sort of research.
None of this excludes trying to improve the source of your ideas, prototyping, talking to users, or fast iteration. All ideas are validated, though. The reason ideas come to your attention and stay there in the first place, among other things, is because they seem upon reflection plausible enough to be a good bet. The point of adding validation to that iteration process explicitly is to increase the thoroughness of your search for existing information that could encourage or discourage your efforts proportional to the costs associated with attempting.
But just because Etsy lacked some better ideas about how to optimize their product doesn’t mean they didn’t get important shit done. Within one year of its initial release, Etsy had gained 10,000 artists (craft makers), pitching 100,000 items, and had a market of approximately 40,000 buyers.
I agree, and that’s why I clarified with
And clearly they were unimaginably successful anyways. Part of my point in the post is to show how poorly you can do things on the “make sure your decisions are correct” axis and still win hard.
I realize how hyperbolic my criticism of Etsy sounds now that I needed this long ass post to explain their judgement errors, but I really think going into this much detail is making the technique seem much more complex than it really is. If you’re planning on spending two months improving the revenue created by feature X by 3%, do the napkin math to see if existing revenue coming from X justifies two months salary. If you’re planning on making a widget that improves the experience of furniture purchasers on your site, make sure there are enough furniture purchasers (taking into consideration the new ones you will be enabling by said widget) to justify maintenance/implementation. According to the guy I linked to, those were the kinds of sanity checks that Etsy wasn’t doing, which personally seems difficult to imagine for people that are otherwise obviously intensely competent.
> If you’re planning on spending two months improving the revenue created by feature X by 3%, do the napkin math to see if existing revenue coming from X justifies two months salary.
How do you know they didn’t? That deck is just a summary of years of work. Perhaps reach out to Dan McKinley for further discussion. I’m afraid we’ve both just making a lot of speculation at this point. Talk to Dan.
When your revenue is 7-8 digits, 3% can add up! Sometimes it’s such a no-brainer that it’s not worth opening Excel over. Last week I had my devs spend a few days implementing a way to create DHL labels using an API. We didn’t do any cost-benefit analysis because it was so clearly something that needed to be done. Today we shipped 147 packages by DHL. That would have taken about 5m each to do the old manual way, so in one day it saved 9.5 hours of staff time. Over one-year that’s an entire salary.
But I think the presentation tells you why they weren’t doing that stuff:
And you know what, if the site’s growth is really insane, it looks like it’s working. You can release things and as long as they don’t completely destroy everything it will look like you’re a genius. All the graphs will go up and to the right.
You can read in the Farnum St. interview with Tobias Lutke, CEO at Shopify, how it held back Shopify’s growth. I’ve met Tobias and he’s a brilliant, level-headed engineer and CEO, but if you read that you’d probably conclude he’s irrational, stupid, and not ambitious enough for not relentlessly maximizing growth for that period. ¯\_(ツ)_/¯
Regarding testing and validation, of course there are exceptions, but generally speaking, it still looks like they got it backwards to me. You’re free to disagree. I’m running multiple companies and have produced many products and services—only one big failure. I’ve tried a lot of approaches and still mix and match many ideas, but can solidly endorse making the prototype in get the feedback.
Maybe you misunderstand how we think about prototyping. A realistic façade is all you need to test with customers. The prototype give you something concrete to put in front of customers for rich feedback and insights. But think lightweight (dirty) version of key aspects of a product or experience. The prototype only needs to be good enough to test out a hypothesis and nothing more. It’s all about testing big ideas with minimal upfront investment.
I recommend the design thinking methodology presented in “Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days.” Invented at Google by Jake Knapp, perfected with more than 150 startups at Google Ventuer (GV).
You need to be a little creative. I’ll give a trivial example that will probably be more enlightening if you went through the proofs C.S. bachelor students have to do of evaluating upper bounds of algorithms.
Lets say you hear about how matrix multiplication is a big problem in ML, and think you have a way to build an algorithm that should give you O(n*sqrt(n)) performance. Your idea happens to be an unusually complicated algorithm and thus a involves a complicated product; you think it will take you around 50 +-20 hours to prototype, and you don’t expect to achieve O(n*sqrt(n)) your first try. You could start your project now and build a prototype as quickly as possible, so that you can get a concrete impression of your algorithms viability. Or, you could first sit and consider for a couple hours and see if there are any ways to test or debunk the feasibility of your algorithm. If you choose the first option, you might get twenty hours in and realize your mistake only then. If you choose the second option, you might realize, just by pondering constraints for an hour, that matrix multiplication requires at least O(n^2) reads, and that your algorithm can’t in any universe work as fast as you expected it to, so no need to build a prototype without further considering where you’re going wrong.
Some ideas aren’t very easily validated. Art (I assume though I am not an artist) is an area where you pretty much just have a difficult to convey imagined finished product, and the best way to start validating is to build it and see if people like it. Lots of seed stage startups are founded to build things no one wants. The reason the team got funding anyways is often because the utility of the product they’re trying to create is difficult to “debunk” from the outset, or fails in hard-to-anticipate ways that aren’t caught by standard sanity checking.
However, outside of a few of those specific cases, most good ideas can be thought about. It might be impossible to completely rule out or in your hunch without building it first. But just by thinking about a proposed solution for the first twelve hours, you can usually establish some sort of probabilistic upper bound on the expected value and lower bound on the cost of implementation/maintenance. For a project that you think is going to take several months, even if you think there’s only a small chance you will figure out a flaw in its viability without prototyping, it’s worth it to sit down and do that sort of research.
None of this excludes trying to improve the source of your ideas, prototyping, talking to users, or fast iteration. All ideas are validated, though. The reason ideas come to your attention and stay there in the first place, among other things, is because they seem upon reflection plausible enough to be a good bet. The point of adding validation to that iteration process explicitly is to increase the thoroughness of your search for existing information that could encourage or discourage your efforts proportional to the costs associated with attempting.
I agree, and that’s why I clarified with
I realize how hyperbolic my criticism of Etsy sounds now that I needed this long ass post to explain their judgement errors, but I really think going into this much detail is making the technique seem much more complex than it really is. If you’re planning on spending two months improving the revenue created by feature X by 3%, do the napkin math to see if existing revenue coming from X justifies two months salary. If you’re planning on making a widget that improves the experience of furniture purchasers on your site, make sure there are enough furniture purchasers (taking into consideration the new ones you will be enabling by said widget) to justify maintenance/implementation. According to the guy I linked to, those were the kinds of sanity checks that Etsy wasn’t doing, which personally seems difficult to imagine for people that are otherwise obviously intensely competent.
> If you’re planning on spending two months improving the revenue created by feature X by 3%, do the napkin math to see if existing revenue coming from X justifies two months salary.
How do you know they didn’t? That deck is just a summary of years of work. Perhaps reach out to Dan McKinley for further discussion. I’m afraid we’ve both just making a lot of speculation at this point. Talk to Dan.
When your revenue is 7-8 digits, 3% can add up! Sometimes it’s such a no-brainer that it’s not worth opening Excel over. Last week I had my devs spend a few days implementing a way to create DHL labels using an API. We didn’t do any cost-benefit analysis because it was so clearly something that needed to be done. Today we shipped 147 packages by DHL. That would have taken about 5m each to do the old manual way, so in one day it saved 9.5 hours of staff time. Over one-year that’s an entire salary.
But I think the presentation tells you why they weren’t doing that stuff:
You can read in the Farnum St. interview with Tobias Lutke, CEO at Shopify, how it held back Shopify’s growth. I’ve met Tobias and he’s a brilliant, level-headed engineer and CEO, but if you read that you’d probably conclude he’s irrational, stupid, and not ambitious enough for not relentlessly maximizing growth for that period. ¯\_(ツ)_/¯
Regarding testing and validation, of course there are exceptions, but generally speaking, it still looks like they got it backwards to me. You’re free to disagree. I’m running multiple companies and have produced many products and services—only one big failure. I’ve tried a lot of approaches and still mix and match many ideas, but can solidly endorse making the prototype in get the feedback.
Maybe you misunderstand how we think about prototyping. A realistic façade is all you need to test with customers. The prototype give you something concrete to put in front of customers for rich feedback and insights. But think lightweight (dirty) version of key aspects of a product or experience. The prototype only needs to be good enough to test out a hypothesis and nothing more. It’s all about testing big ideas with minimal upfront investment.
I recommend the design thinking methodology presented in “Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days.” Invented at Google by Jake Knapp, perfected with more than 150 startups at Google Ventuer (GV).
Prototype comes before Test.
https://www.thesprintbook.com/how
Cheers,