Thanks for pointing this out.
Let’s use Beeminder as an example. When I emailed Daniel he said this: “we’ve talked with the CFAR founders in the past about setting up RCTs for measuring the effectiveness of beeminder itself and would love to have that see the light of day”.
Which is a little open ended, so I’m going to arbitrarily decide that we’ll study Beeminder for weight loss effectiveness.
Story* as follows:
Daniel goes to (our thing).com and registers a new study. He agrees to the terms, and tells us that this is a study which can impact health—meaning that mandatory safety questions will be required. Once the trial is registered it is viewable publicly as “initiated”.
He then takes whatever steps we decide on to locate participants. Those participants are randomly assigned to two groups: (1) act normal, and (2) use Beeminder to track exercise and food intake. Every day the participants are sent a text message with a URL where they can log that day’s data. They do so.
After two weeks, the study completes and both Daniel and the world are greeted with the results. Daniel can now update Beeminder.com to say that Beeminder users lost XY pounds more than the control group… and when a rationalist sees such claims they can actually believe them.
Note that this story isn’t set in stone—just a sketch to aid discussion
In my experience, startups want to demonstrate efficacy about basic things: weight removed, increased revenue, personal productivity, product safety, etc.
This kind of research lends itself extremely well to protocol templates: a standardized sequence of steps to locate the participants, collect the data, and decide the results. These steps could be performed by a website. I’ve posted a story of how that might work here.
Without such a project, founders have two options:
Perform the study themselves. The scientific background and time required to design a study well is non-trivial, as is the expertise necessary to create participant waivers. There are also significant time costs in setting up the data collection, and performing the analysis. At the end, the study will be greeted somewhat skeptically.
Pay a University or Contract Research Organization to perform the study. This is expensive, which I believe is why more founders aren’t doing it.
This project creates a third option:
Use a standard template then get back to work.
Which may be imperfect, but is still a pretty appealing value proposition.