Regarding the way Etsy was prioritizing development, it sounds like even their late stage “idea > validate > prototype” cycle is wrong. How can you “validate” before you have a prototype to get feedback on? Where did the idea come from? I’d recommend starting with customer discovery talks. Read “The Mom Test” to learn how to talk to customers about their problems. Then you can take those ideas into a design sprint to mock-up and prototype a feature so that you have something you can get feedback on.
But just because Etsy lacked some better ideas about how to optimize their product doesn’t mean they didn’t get important shit done. Within one year of its initial release, Etsy had gained 10,000 artists (craft makers), pitching 100,000 items, and had a market of approximately 40,000 buyers.
Etsy as an organization, did not care or want to be correct at predicting what its users would want,
In a word, I’d say it’s probably not that Etsy didn’t care about how to do these things better. The person in charge of development probably just didn’t know how to do these things better. I’m certainly doing things differently today than I did 5 years ago. We live, we learn. It’s a process.
Regarding learning a foreign language, I’m not sure what I can say. I speak Japanese, and I run a company which makes some top selling Japanese language learning products. So I know something about this topic. You’re right, learning a foreign language is a big commitment. So isn’t it obvious that the longer the required commitment, the most likely it is that people will drop out? The same is true for college:
According to College Atlas, 70% of Americans will study at a four-year college, but less than two-thirds will graduate with a degree, and 30% of first-year students drop out after their first year of school.
In the case of learning a foreign language, maybe over time the quitter just decided that the effort was no longer worth it. Maybe that want to prioritize other hobbies and interests. Or maybe they just don’t enjoy memorizing hanzi. Maybe the idea moving to China or the important of talking to people in Chinese has lost its luster. What’s wrong with changing your mind about these things?
If you study a foreign language because you think it will be useful or cool, and then don’t end up speaking it to a degree that makes up for the time invested, then yes, that is a mistake in hindsight.
That’s one way to look at it. But could you been letting sunk costs fallacy get the best of you? Here’s another way to look at it: what if our beginner or intermediate student of Chinese decides that speaking Chinese is no longer an important goals of theirs? From that point on, even moment they continue to spend learning Chinese is a waste of time and a mistake.
Also, maybe a lot of people don’t realize how much of a commitment learning a language really is. The internet is full misleading information and false promises perpetuated by get-rich-quick schemes and douche bags like Tai Lopez and Jim Kwik who try to separate people from their money at an industrial-scale with claims like “how I learned 5 languages in a year” and promises that you too can “take your dream trip and actually be able to speak fluently” normally for 5 easy payments of $79.99, now for the low price of $39.95! ;)
My point with the anecdote is that much of the time this is nonsense. I spent so much time twisting my brain into an M.C. Escher painting to try and push myself to take actions I knew were good for me, that I often forgot to try honestly bringing the costs and benefits of the thing to the forefront of my mind and seeing if that would be enough. It turns out, quite often enough it is, and even further sometimes I realize that I don’t actually value the thing I’m trying to convince myself to do enough to do it.
This sounds like valuable progress in your thinking! Probably just writing about it has helped you see things more clear. I hope you’ll go back to my edited post. I shared some ideas about starting a journaling habit for precisely this reason!
Regarding learning a foreign language, I’m not sure what I can say. I speak Japanese, and I run a company which makes some top selling Japanese language learning products. So I know something about this topic. You’re right, learning a foreign language is a big commitment. So isn’t it obvious that the longer the required commitment, the most likely it is that people will drop out?
Yes. It’s very obvious. Which is why it’s that much more unusual that people make these commitments without more consideration than they tend to put forward. People will intuit that they should do some significant (meaning more than a couple hours) hard research into whether or not it’s worth it to get their masters, and where to get it; they tend not to realize that successfully learning a language as remote as Mandarin requires similar time investment as getting the masters degree, even though a necessary part of any plan to do so would be to figure that out, and I don’t know why. I suspect that many of these people are just going through the motions with no real objectives in mind, and that’s a running theme of my OP.
In the case of learning a foreign language, maybe over time the quitter just decided that the effort was no longer worth it. Maybe that want to prioritize other hobbies and interests. Or maybe they just don’t enjoy memorizing hanzi. Maybe the idea moving to China or the important of talking to people in Chinese has lost its luster. What’s wrong with changing your mind about these things?
As you say, nothing. The problem, at least in hindsight, was the decision to start in the first place. You can say the decision was correct given the information they had, but I’m trying to make the point that much (most?) of the time this is clearly not the case, and people engage in studying habits that will not ensure they learn the language in the next forty years.
Which is not to say that people shouldn’t use your product or service. I personally think language acquisition has some pretty significant positive externalities—languages are more useful the more people speak them, after all—but we don’t even need to go there. It’s just that the way most people set about learning them often implies they’re not even trying, to a degree that’s difficult to make sense of, at least for me.
This sounds like valuable progress in your thinking! Probably just writing about it has helped you see things more clear. I hope you’ll go back to my edited post. I shared some ideas about starting a journaling habit for precisely this reason!
Regarding the way Etsy was prioritizing development, it sounds like even their late stage “idea > validate > prototype” cycle is wrong. How can you “validate” before you have a prototype to get feedback on?
You need to be a little creative. I’ll give a trivial example that will probably be more enlightening if you went through the proofs C.S. bachelor students have to do of evaluating upper bounds of algorithms.
Lets say you hear about how matrix multiplication is a big problem in ML, and think you have a way to build an algorithm that should give you O(n*sqrt(n)) performance. Your idea happens to be an unusually complicated algorithm and thus a involves a complicated product; you think it will take you around 50 +-20 hours to prototype, and you don’t expect to achieve O(n*sqrt(n)) your first try. You could start your project now and build a prototype as quickly as possible, so that you can get a concrete impression of your algorithms viability. Or, you could first sit and consider for a couple hours and see if there are any ways to test or debunk the feasibility of your algorithm. If you choose the first option, you might get twenty hours in and realize your mistake only then. If you choose the second option, you might realize, just by pondering constraints for an hour, that matrix multiplication requires at least O(n^2) reads, and that your algorithm can’t in any universe work as fast as you expected it to, so no need to build a prototype without further considering where you’re going wrong.
Some ideas aren’t very easily validated. Art (I assume though I am not an artist) is an area where you pretty much just have a difficult to convey imagined finished product, and the best way to start validating is to build it and see if people like it. Lots of seed stage startups are founded to build things no one wants. The reason the team got funding anyways is often because the utility of the product they’re trying to create is difficult to “debunk” from the outset, or fails in hard-to-anticipate ways that aren’t caught by standard sanity checking.
However, outside of a few of those specific cases, most good ideas can be thought about. It might be impossible to completely rule out or in your hunch without building it first. But just by thinking about a proposed solution for the first twelve hours, you can usually establish some sort of probabilistic upper bound on the expected value and lower bound on the cost of implementation/maintenance. For a project that you think is going to take several months, even if you think there’s only a small chance you will figure out a flaw in its viability without prototyping, it’s worth it to sit down and do that sort of research.
None of this excludes trying to improve the source of your ideas, prototyping, talking to users, or fast iteration. All ideas are validated, though. The reason ideas come to your attention and stay there in the first place, among other things, is because they seem upon reflection plausible enough to be a good bet. The point of adding validation to that iteration process explicitly is to increase the thoroughness of your search for existing information that could encourage or discourage your efforts proportional to the costs associated with attempting.
But just because Etsy lacked some better ideas about how to optimize their product doesn’t mean they didn’t get important shit done. Within one year of its initial release, Etsy had gained 10,000 artists (craft makers), pitching 100,000 items, and had a market of approximately 40,000 buyers.
I agree, and that’s why I clarified with
And clearly they were unimaginably successful anyways. Part of my point in the post is to show how poorly you can do things on the “make sure your decisions are correct” axis and still win hard.
I realize how hyperbolic my criticism of Etsy sounds now that I needed this long ass post to explain their judgement errors, but I really think going into this much detail is making the technique seem much more complex than it really is. If you’re planning on spending two months improving the revenue created by feature X by 3%, do the napkin math to see if existing revenue coming from X justifies two months salary. If you’re planning on making a widget that improves the experience of furniture purchasers on your site, make sure there are enough furniture purchasers (taking into consideration the new ones you will be enabling by said widget) to justify maintenance/implementation. According to the guy I linked to, those were the kinds of sanity checks that Etsy wasn’t doing, which personally seems difficult to imagine for people that are otherwise obviously intensely competent.
> If you’re planning on spending two months improving the revenue created by feature X by 3%, do the napkin math to see if existing revenue coming from X justifies two months salary.
How do you know they didn’t? That deck is just a summary of years of work. Perhaps reach out to Dan McKinley for further discussion. I’m afraid we’ve both just making a lot of speculation at this point. Talk to Dan.
When your revenue is 7-8 digits, 3% can add up! Sometimes it’s such a no-brainer that it’s not worth opening Excel over. Last week I had my devs spend a few days implementing a way to create DHL labels using an API. We didn’t do any cost-benefit analysis because it was so clearly something that needed to be done. Today we shipped 147 packages by DHL. That would have taken about 5m each to do the old manual way, so in one day it saved 9.5 hours of staff time. Over one-year that’s an entire salary.
But I think the presentation tells you why they weren’t doing that stuff:
And you know what, if the site’s growth is really insane, it looks like it’s working. You can release things and as long as they don’t completely destroy everything it will look like you’re a genius. All the graphs will go up and to the right.
You can read in the Farnum St. interview with Tobias Lutke, CEO at Shopify, how it held back Shopify’s growth. I’ve met Tobias and he’s a brilliant, level-headed engineer and CEO, but if you read that you’d probably conclude he’s irrational, stupid, and not ambitious enough for not relentlessly maximizing growth for that period. ¯\_(ツ)_/¯
Regarding testing and validation, of course there are exceptions, but generally speaking, it still looks like they got it backwards to me. You’re free to disagree. I’m running multiple companies and have produced many products and services—only one big failure. I’ve tried a lot of approaches and still mix and match many ideas, but can solidly endorse making the prototype in get the feedback.
Maybe you misunderstand how we think about prototyping. A realistic façade is all you need to test with customers. The prototype give you something concrete to put in front of customers for rich feedback and insights. But think lightweight (dirty) version of key aspects of a product or experience. The prototype only needs to be good enough to test out a hypothesis and nothing more. It’s all about testing big ideas with minimal upfront investment.
I recommend the design thinking methodology presented in “Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days.” Invented at Google by Jake Knapp, perfected with more than 150 startups at Google Ventuer (GV).
Regarding the way Etsy was prioritizing development, it sounds like even their late stage “idea > validate > prototype” cycle is wrong. How can you “validate” before you have a prototype to get feedback on? Where did the idea come from? I’d recommend starting with customer discovery talks. Read “The Mom Test” to learn how to talk to customers about their problems. Then you can take those ideas into a design sprint to mock-up and prototype a feature so that you have something you can get feedback on.
But just because Etsy lacked some better ideas about how to optimize their product doesn’t mean they didn’t get important shit done. Within one year of its initial release, Etsy had gained 10,000 artists (craft makers), pitching 100,000 items, and had a market of approximately 40,000 buyers.
In a word, I’d say it’s probably not that Etsy didn’t care about how to do these things better. The person in charge of development probably just didn’t know how to do these things better. I’m certainly doing things differently today than I did 5 years ago. We live, we learn. It’s a process.
Regarding learning a foreign language, I’m not sure what I can say. I speak Japanese, and I run a company which makes some top selling Japanese language learning products. So I know something about this topic. You’re right, learning a foreign language is a big commitment. So isn’t it obvious that the longer the required commitment, the most likely it is that people will drop out? The same is true for college:
In the case of learning a foreign language, maybe over time the quitter just decided that the effort was no longer worth it. Maybe that want to prioritize other hobbies and interests. Or maybe they just don’t enjoy memorizing hanzi. Maybe the idea moving to China or the important of talking to people in Chinese has lost its luster. What’s wrong with changing your mind about these things?
That’s one way to look at it. But could you been letting sunk costs fallacy get the best of you? Here’s another way to look at it: what if our beginner or intermediate student of Chinese decides that speaking Chinese is no longer an important goals of theirs? From that point on, even moment they continue to spend learning Chinese is a waste of time and a mistake.
Also, maybe a lot of people don’t realize how much of a commitment learning a language really is. The internet is full misleading information and false promises perpetuated by get-rich-quick schemes and douche bags like Tai Lopez and Jim Kwik who try to separate people from their money at an industrial-scale with claims like “how I learned 5 languages in a year” and promises that you too can “take your dream trip and actually be able to speak fluently” normally for 5 easy payments of $79.99, now for the low price of $39.95! ;)
This sounds like valuable progress in your thinking! Probably just writing about it has helped you see things more clear. I hope you’ll go back to my edited post. I shared some ideas about starting a journaling habit for precisely this reason!
Yes. It’s very obvious. Which is why it’s that much more unusual that people make these commitments without more consideration than they tend to put forward. People will intuit that they should do some significant (meaning more than a couple hours) hard research into whether or not it’s worth it to get their masters, and where to get it; they tend not to realize that successfully learning a language as remote as Mandarin requires similar time investment as getting the masters degree, even though a necessary part of any plan to do so would be to figure that out, and I don’t know why. I suspect that many of these people are just going through the motions with no real objectives in mind, and that’s a running theme of my OP.
As you say, nothing. The problem, at least in hindsight, was the decision to start in the first place. You can say the decision was correct given the information they had, but I’m trying to make the point that much (most?) of the time this is clearly not the case, and people engage in studying habits that will not ensure they learn the language in the next forty years.
Which is not to say that people shouldn’t use your product or service. I personally think language acquisition has some pretty significant positive externalities—languages are more useful the more people speak them, after all—but we don’t even need to go there. It’s just that the way most people set about learning them often implies they’re not even trying, to a degree that’s difficult to make sense of, at least for me.
thought of you!
https://www.youtube.com/watch?v=lhcvejeAB0E
Perhaps >.>
You need to be a little creative. I’ll give a trivial example that will probably be more enlightening if you went through the proofs C.S. bachelor students have to do of evaluating upper bounds of algorithms.
Lets say you hear about how matrix multiplication is a big problem in ML, and think you have a way to build an algorithm that should give you O(n*sqrt(n)) performance. Your idea happens to be an unusually complicated algorithm and thus a involves a complicated product; you think it will take you around 50 +-20 hours to prototype, and you don’t expect to achieve O(n*sqrt(n)) your first try. You could start your project now and build a prototype as quickly as possible, so that you can get a concrete impression of your algorithms viability. Or, you could first sit and consider for a couple hours and see if there are any ways to test or debunk the feasibility of your algorithm. If you choose the first option, you might get twenty hours in and realize your mistake only then. If you choose the second option, you might realize, just by pondering constraints for an hour, that matrix multiplication requires at least O(n^2) reads, and that your algorithm can’t in any universe work as fast as you expected it to, so no need to build a prototype without further considering where you’re going wrong.
Some ideas aren’t very easily validated. Art (I assume though I am not an artist) is an area where you pretty much just have a difficult to convey imagined finished product, and the best way to start validating is to build it and see if people like it. Lots of seed stage startups are founded to build things no one wants. The reason the team got funding anyways is often because the utility of the product they’re trying to create is difficult to “debunk” from the outset, or fails in hard-to-anticipate ways that aren’t caught by standard sanity checking.
However, outside of a few of those specific cases, most good ideas can be thought about. It might be impossible to completely rule out or in your hunch without building it first. But just by thinking about a proposed solution for the first twelve hours, you can usually establish some sort of probabilistic upper bound on the expected value and lower bound on the cost of implementation/maintenance. For a project that you think is going to take several months, even if you think there’s only a small chance you will figure out a flaw in its viability without prototyping, it’s worth it to sit down and do that sort of research.
None of this excludes trying to improve the source of your ideas, prototyping, talking to users, or fast iteration. All ideas are validated, though. The reason ideas come to your attention and stay there in the first place, among other things, is because they seem upon reflection plausible enough to be a good bet. The point of adding validation to that iteration process explicitly is to increase the thoroughness of your search for existing information that could encourage or discourage your efforts proportional to the costs associated with attempting.
I agree, and that’s why I clarified with
I realize how hyperbolic my criticism of Etsy sounds now that I needed this long ass post to explain their judgement errors, but I really think going into this much detail is making the technique seem much more complex than it really is. If you’re planning on spending two months improving the revenue created by feature X by 3%, do the napkin math to see if existing revenue coming from X justifies two months salary. If you’re planning on making a widget that improves the experience of furniture purchasers on your site, make sure there are enough furniture purchasers (taking into consideration the new ones you will be enabling by said widget) to justify maintenance/implementation. According to the guy I linked to, those were the kinds of sanity checks that Etsy wasn’t doing, which personally seems difficult to imagine for people that are otherwise obviously intensely competent.
> If you’re planning on spending two months improving the revenue created by feature X by 3%, do the napkin math to see if existing revenue coming from X justifies two months salary.
How do you know they didn’t? That deck is just a summary of years of work. Perhaps reach out to Dan McKinley for further discussion. I’m afraid we’ve both just making a lot of speculation at this point. Talk to Dan.
When your revenue is 7-8 digits, 3% can add up! Sometimes it’s such a no-brainer that it’s not worth opening Excel over. Last week I had my devs spend a few days implementing a way to create DHL labels using an API. We didn’t do any cost-benefit analysis because it was so clearly something that needed to be done. Today we shipped 147 packages by DHL. That would have taken about 5m each to do the old manual way, so in one day it saved 9.5 hours of staff time. Over one-year that’s an entire salary.
But I think the presentation tells you why they weren’t doing that stuff:
You can read in the Farnum St. interview with Tobias Lutke, CEO at Shopify, how it held back Shopify’s growth. I’ve met Tobias and he’s a brilliant, level-headed engineer and CEO, but if you read that you’d probably conclude he’s irrational, stupid, and not ambitious enough for not relentlessly maximizing growth for that period. ¯\_(ツ)_/¯
Regarding testing and validation, of course there are exceptions, but generally speaking, it still looks like they got it backwards to me. You’re free to disagree. I’m running multiple companies and have produced many products and services—only one big failure. I’ve tried a lot of approaches and still mix and match many ideas, but can solidly endorse making the prototype in get the feedback.
Maybe you misunderstand how we think about prototyping. A realistic façade is all you need to test with customers. The prototype give you something concrete to put in front of customers for rich feedback and insights. But think lightweight (dirty) version of key aspects of a product or experience. The prototype only needs to be good enough to test out a hypothesis and nothing more. It’s all about testing big ideas with minimal upfront investment.
I recommend the design thinking methodology presented in “Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days.” Invented at Google by Jake Knapp, perfected with more than 150 startups at Google Ventuer (GV).
Prototype comes before Test.
https://www.thesprintbook.com/how
Cheers,