DavidLS

• DavidLSOct 19, 2014, 10:37 PM

  2 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Yes. I think if we can manage it, requiring data-analysis to be pre-declared is just better. I don’t think science as a whole can do this, because not all data is as cheap to produce as product testing data.

  Now that I’ve heard your reply to question #8, I need to consider this again. Perhaps we could have some basic claims done by software, while allowing for additional claims such as “those over 50 show twice the results” to be verified by grad students. I will think about this.

• DavidLSOct 19, 2014, 10:25 PM

  2 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  FTC is so much better than lawsuit. I don’t know a single advertiser that isn’t afraid of the FTC. It looks like enforcement is tied to complaint numbers, so the press release should include information about how to personally complain (and go out to a mailing list as well).

  I would love assistance with the agreements. It sounds like you would be more suited to the Business <> Non-Profit agreements than the Participant <> Business agreements. How do I maximize the value of your contribution? Are you more suited to the high-level term sheet, or the final wording?

• DavidLSOct 19, 2014, 10:22 PM

  2 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  I’m very glad I asked for more clarification. I’m going to call this system The Reviewer’s Dilemma, it’s a very interesting solution for allowing non-software analysis to occur in a trusted manner. I am somewhat worried about a laziness bias (it’s much easier to agree than disprove), but I imagine that there is a similar bounty for overturning previous results this might be handled.

  I’ll do a little customer development with some friends, but the possibility of reviewers being added as co-authors might also act as a nice incentive (both to reduce laziness, and as addition compensation).

• DavidLSOct 19, 2014, 9:59 PM

  4 points

  in reply to: ChristianKl’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Thank you. I had not seen the reproducibility initiative. Link very much appreciated, I’ll start the conversation tonight. PLoS hosting the application would be ideal.

• DavidLSOct 19, 2014, 7:44 AM

  7 points

  in reply to: Gunnar_Zarncke’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Oh, interesting.

  I had been assuming that participants needed to be drawn from the general population. If we don’t think there’s too much hazard there, I agree a points system would work. Some portion of the population would likely just enjoy the idea of receiving free product to test.

• DavidLSOct 19, 2014, 7:31 AM

  1 point

  in reply to: blogospheroid’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Your approach to blinding makes sense, and works. I thought we were trying for a zero third party approach though?

  I was giving more thought to a distributed solution during dinner, and I think I see how to solve the physical shipments problem in a scalable way. I’m still not 100% sold on it, but consider these two options:

  • You ship both a placebo package and a non-placebo package to the participant, and have them flip a coin to decide which one to use. They either throw away or disregard the other package for the duration of the study.

  • You ship N packages to Total/​N participants. The participants which receive N packages then randomly assigns himself a package, and randomly distributes the remaining (N-1) packages to other participants.

  They both require trusting the participant with assignment. Which feels wrong to me, but I’m not sure why...

• DavidLSOct 19, 2014, 7:05 AM

  4 points

  in reply to: Metus’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Thank you. I had forgotten about that.

  So let’s say the two groups were, as you suggest:

  • Tracking food & exercise on Beeminder

  • Tracking food & exercise in a journal

  Do you have any thoughts on what questions we should be asking about this product? Somehow the data collection and analysis once we have the timeseries data doesn’t seem so hard… but the protocol and question design seems very difficult to me.

• DavidLSOct 19, 2014, 6:24 AM

  3 points

  in reply to: Metus’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Thank you. Help considering the methodology and project growth prospects is very much appreciated.

  I agree that compatibility with traditional papers is important. It was not stated explicitly, but I do want the results to be publishable in traditional journals. I plan on publishing the results for my company’s product. It seemed to me like being overly rigorous might be a selling point initially—“sure we did the study cheap /​ didn’t use a university, but look how insanely rigorous we were”

  Going after professional researchers seems much harder. They actually know how to perform the research, so the value proposition is much weaker—they are already trusted, and know how to use R :p

  These are just initial thoughts. I’ll think about this more.

• DavidLSOct 19, 2014, 6:05 AM

  3 points

  in reply to: Nightspacer’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Thanks, fixed.

• DavidLSOct 19, 2014, 3:22 AM

  6 points

  in reply to: blogospheroid’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Interesting. I’m hoping that by getting a trustworthy non-profit to host the site (and paying for a security audit) we can largely side step the issues.

  I spent a long time trying to create a way not to need the trusted third party, but I kept hitting dead ends. The specific dead end that hurt the most was blinding of physical product shipments.

  If we can figure out a way to ship both products and placebos to people without knowing who’s getting what, I think we can do this :)

• DavidLSOct 19, 2014, 1:38 AM

  4 points

  in reply to: Lumifer’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  This is not at all self-evident to me. How, for example, would you demonstrate product safety (for a diverse variety of products) via a standard template?

  Templates not template. I think if you know roughly which bodily systems a product is likely to effect, the questions are not so diverse.

  My background is not in question selection (it’s ML and webapp programming), but here goes some general question ideas for edible products:

  • I have/​have not felt sick to my stomach in the last 24 hours. (standardized 1-7 to rate severity)

  • I have/​have not felt dizzy in the last 24 hours. (standardized 1-7 to rate severity)

  • Bristol stool scale score

  The mandatory questions are intended to give LessWrong /​ everyone a say in what startups will test their products for—NOT to provide a 100% guarantee of general safety (the FDA already handles that). We should use these questions to learn about unanticipated side effects.

  Research costs money and requires competent people. If it were possible to do meaningful research on the cheap just by reusing the same template, don’t you think it would be a very popular opinion already?

  I’m hope it will do something akin to what Google Translate did for translation: lower the cost for modest use cases. If you want a high quality translation (poetry) you still need to hire a good translator. However, if you are willing to accept a reasonably good level of translation quality, it’s now free.

  I agree it’s weird that somebody else hasn’t noticed. testifiable.com is the closest I’ve found. I’ve already spoken with Testifiable founder’s and invited them to this thread.

  What links here?

  • DavidLS's comment on Fix­ing Mo­ral Hazards In Busi­ness Science by DavidLS (Oct 21, 2014, 11:32 PM; 0 points)

• DavidLSOct 19, 2014, 1:08 AM

  5 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  8 - Have you considered some form of reputation system, allowing commenters to build a reputation for debunking badly supported claims and affirming well-supported claims? (Or perhaps some other goodie?) I can imagine it becoming a pastime for grad students, which would be a Good Thing (TM).

  I hadn’t. I like the idea, but am less able to visualize it than the rest of this stuff. Grad students cleaning up marketing claims does indeed sound like a Good Thing...

• DavidLSOct 19, 2014, 1:08 AM

  11 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  7 - Can sponsors do a private mini-trial to test its trial design before going full bore (presumably, with their promise not to publicize the results)?

  This is an awesome idea. I had not considered this until you posted it. This sounds great.

• DavidLSOct 19, 2014, 1:07 AM

  7 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  6 - Does a sponsor have any recourse if it designed the trial badly, leading to misleading results? Or is its remedy really to design a better trial and publicize that one?

  This is a hard one. I anticipate that at least initially only Good People will be using this protocol. These are people who spent a lot of time creating something to (hopefully) make the world better. Not cool to screw them if they make a mistake, or if v1 isn’t as awesome as anticipated.

  A related question is: what can we do to help a company that has demonstrated its effectiveness?

• DavidLSOct 19, 2014, 1:06 AM

  5 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  5 - What do the participants get? Is that simply up to the sponsor? If so, who reviews it to assure that the incentive does not distort the data? If no one, will you at least require that the incentive be reported as part of the trial?

  We need to design rules governing participant compensation.

  At a minimum I think all compensation should be reported (it’s part of what’s needed for replication), and of course not related to the results a participant reports. Ideally we create a couple defined protocols for locating participants, and people largely choose to go with a known good solution.

• DavidLSOct 19, 2014, 1:05 AM

  5 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  4 - What action does the organization behind the app take if a sponsor publicly misrepresents the data or, more likely, its meaning? If the organization would take action, does it take the same action if the statement is merely misleading, rather than factually incorrect?

  I imagined similar actions as the Free Software Foundation takes when a company violates the GPL: basically a lawsuit and press release warning people. For template studies, ideally what claims can be made would be specified by the template (ie “Our users lost XY more pounds over Z time”.)

• DavidLSOct 19, 2014, 1:04 AM

  7 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  3 - If the sponsoring company does its own math and stats, must it publicly post its working papers before making claims based on the data? Does anyone review that to make sure it passes some light smell test, and isn’t just pictures of cats?

  At minimum the code used should be posted publicly and open-source licensed (otherwise there can be no scrutiny or replication). I also think paying to have a third party review the code isn’t unreasonable.

• DavidLSOct 19, 2014, 1:03 AM

  7 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  2 - Is the data (presumably anonymized) made publicly available, so that others can dispute the meaning?

  That was the initial plan, yes! Beltran (my co-founder at GB) is worried that will result in either HIPPA issues or something like this, so I’m ultimately unsure. Putting structures in place so the science is right the first time seems better.

• DavidLSOct 19, 2014, 1:01 AM

  5 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  1 - For more complicated propositions, who does the math and statistics? The application apparently gathers the data, but it is still subject to interpretation.

  This problem can be reduced in size by having the webapp give out blinded data, and only reveal group names after the analysis has been publicly committed to. If participating companies are unhappy with the existing modules, they could perhaps hire “statistical consultants” to add a module, permanently improving the site for everyone.

  This could be related to your #8 as well :)

  What links here?

  • DavidLS's comment on Fix­ing Mo­ral Hazards In Busi­ness Science by DavidLS (Oct 20, 2014, 1:55 AM; 3 points)

• DavidLSOct 19, 2014, 12:59 AM

  5 points

  in reply to: MaximumLiberty’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Thank you! This is exactly the kind of discussion I was hoping for.

  The general answer to your questions is: I want to build whatever LessWrong wants me to build. If it’s debated in the open, and agreed as the least-worst option, that’s the plan.

  I’ll post answers to each question in a separate thread, since they raise a lot of questions I was hoping for feedback on.