EY endorsed it strongly (which I believe counts as a ding on his track record if anyone is maintaining that anywhere)
I don’t think it’s a ding on his track record.
He tried a product.
It worked shockingly well for him.
He recommended others use that product.
This is a basic prosocial act. You haven’t made an argument that the product was low-quality, the failure of the company only shows that there wasn’t enough of a market for that particular product to sustain the company. For the most part I’m glad Eliezer advertised it while I could still buy it, it seems like the product was pretty great (though very expensive).
For context, here is an advert he gave for them, which has no endorsement of funding the company, nor says the organization is well run, and entirely focuses on his experience of the product (with the exception of one parenthetical).
You know Harry’s non-24 sleep disorder? I have that. Normally my days are around 24 hours and 30 minutes long.
Around a year ago, some friends of mine cofounded MetaMed, intended to provide high-grade analysis of the medical literature for people with solution-resistant medical problems. (I.e. their people know Bayesian statistics and don’t automatically believe every paper that claims to be ‘statistically significant’ – in a world where only 20-30% of studies replicate, they not only search the literature, but try to figure out what’s actually true.) MetaMed offered to demonstrate by tackling the problem of my ever-advancing sleep cycle.
Here’s some of the things I’ve previously tried:
Taking low-dose melatonin 1-2 hours before bedtime
Using timed-release melatonin
Installing red lights (blue light tells your brain not to start making melatonin)
Using blue-blocking sunglasses after sunset
Wearing earplugs
Using a sleep mask
Watching the sunrise
Watching the sunset
Blocking out all light from the windows in my bedroom using aluminum foil, then lining the door-edges with foam to prevent light from slipping in the cracks, so I wouldn’t have to use a sleep mask
Spending a total of ~$2200 on three different mattresses (I cannot afford the high-end stuff, so I tried several mid-end ones)
Trying 4 different pillows, including memory foam, and finally settling on a folded picnic blanket stuffed into a pillowcase (everything else was too thick)
Putting 2 humidifiers in my room, a warm humidifier and a cold humidifier, in case dryness was causing my nose to stuff up and thereby diminish sleep quality
Buying an auto-adjusting CPAP machine for $650 off Craigslist in case I had sleep apnea. ($650 is half the price of the sleep study required to determine if you need a CPAP machine.)
Taking modafinil and R-modafinil.
Buying a gradual-light-intensity-increasing, sun alarm clock for ~$150
Not all of this was futile – I kept the darkened room, the humidifiers, the red lights, the earplugs, and one of the mattresses; and continued taking the low-dose and time-release melatonin. But that didn’t prevent my sleep cycle from advancing 3 hours per week (until my bedtime was after sunrise, whereupon I would lose several days to staying awake until sunset, after which my sleep cycle began slowly advancing again).
MetaMed produced a long summary of extant research on non-24 sleep disorder, which I skimmed, and concluded by saying that – based on how the nadir of body temperature varies for people with non-24 sleep disorder and what this implied about my circadian rhythm – their best suggestion, although it had little or no clinical backing, was that I should take my low-dose melatonin 5-7 hours before bedtime, instead of 1-2 hours, a recommendation which I’d never heard anywhere before.
And it worked.
I can’t *#&$ing believe that #*$%ing worked.
(EDIT in response to reader questions: “Low-dose” melatonin is 200microgram (mcg) = 0.2 mg. Currently I’m taking 0.2mg 5.5hr in advance, and taking 1mg timed-release just before closing my eyes to sleep. However, I worked up to that over time – I started out just taking 0.3mg total, and I would recommend to anyone else that they start at 0.2mg.)
Sticker shock warning: MetaMed’s charge for an analysis starts at $5K, or around double the cost of everything else I tried put together – it’s either for people who have money, or people who have resistant serious problems. (Of course MetaMed dreams of eventually converting all of medicine to a saner footing, but right now they have to charge significant amounts to initial customers.) And by the nature of MetaMed’s task, results are definitely not guaranteed – but it worked for me.
...there’s also MetaMed. Instead of just having “evidence-based medicine” in journals that doctors don’t actually read, MetaMed will provide you with actual evidence-based healthcare.
You’re right that he doesn’t make any specific verifiable claims so much as be very glowing and excited. It does still make me less inclined to trust his predictive ability (or trust him, depending on how much is him believing in that stuff vs building up hype for whatever reason.)
I do think this ad doesn’t line up with what you said re: “[...] nor says the organization is well run, and entirely focuses on his experience of the product (with the exception of one parenthetical).”
As I understand it, you’re updating against his recommendations of a product by his friends being strong evidence that the company won’t later go out of business. This seems fine to me.
I’m saying that his endorsement of the product seems eminently reasonable to me, that it was indeed life-changing for him on a level that very few products ever are, and that in general with that kind of information about a product, I don’t think he made any errors of judgment, and acted pro-socially.
I will continue to take his product advice strongly, but I will not expect that just because a company is run by rationalists or that Eliezer endorses the product, that this is especially strong evidence that they will succeed on the business fundamentals.
I think you were mistaken to call it a “ding on his track record” because he did not endorse investing in the company, he endorsed using the product, and this seems like the right epistemic state to me. From the evidence I have about MetaMed, I would really want to have access to their product.
As an example, if he’d written a post called “Great Investment Opportunity: MetaMed” this would be a ding on his track record. Instead he wrote a post called “MetaMed: Evidence-Based Healthcare”, and this seems accurate and to be a positive sign about his track record of product-recommendations.
This is a new service and it has to interact with the existing medical system, so they are currently expensive, starting at $5,000 for a research report. (Keeping in mind that a basic report involves a lot of work by people who must be good at math.)
Unrelatedly but from the same advert. I had not realized it was that expensive—this rings some alarm bells for me but maybe it is fine, it is in fact a medical service. I have been waffling back and forth and will conclude I don’t know enough of the details.
Regardless, the alarm bells still made me want to survey the comments and see if anyone else was alarmed. Summaries of the comments by top level:
> The words “evidence-based medicine” seems to imply “non evidence-based medicine”
> Will MetaMed make its research freely available?
> Proposals re: the idea that MetaMed might not improve the world save for their clients
> You should disclose that MIRI shares sponsors with MetaMed, detail question
> Please send this to the front page!
> I’m overall not impressed, here are a couple criticisms, what does MetaMed have over uptodate.com in terms of comparative advantage? (Nice going user EHeller, have some Bayes points.)
> Discussion of doctors and their understanding of probability
> MetaMed has gone out of business (3 years later)
> Is MetaMed a continuation of a vanished company called Personalized Medicine?
> A friend of mine has terrible fibromyalgia and would pay 5k for relief but not for a literature search of unknown benefit. I guess she’s not the target audience? (long thread, MetaMed research is cited, EHeller again disputes its value compared to less expensive sources)
> An aside on rat poison
> How might MetaMed and IBM Watson compare and contrast?
> Error in advert: Jaan Tallinn is not the CEO but chairman, Zvi is the CEO.
> Is MetaMed LW-y enough that we should precommit to updating by prespecified amounts on the effectiveness of LW rationality in response to its successes and failures?
There I will cut off because the last commentor is after my own heart. Gwern responds by saying:
At a first glance, I’m not sure humans can update by prespecified amounts, much less prespecified amounts of the right quantity in this case: something like >95% of all startups fail for various reasons, so even if LW-think could double the standard odds (let’s not dicker around with merely increasing effectiveness by 50% or something, let’s go all the way to +100%!), you’re trying to see the difference between… a 5% success rate and a 10% success rate. One observation just isn’t going to count for much here.
And that is correct. But you don’t have to make a single prediction, success/fail, you should be able to come up with predictions about your company that you can put higher numbers on and we can see how those empirically turn out. Or you could even keep track of all the startups launched by prominent LW members.
In contrast, Michael Vassar (who was also on the project) says,
Definitely, though others must decide the update size.
Which I don’t think anyone followed through on, perhaps because they then agreed with gwern?
Anyway—it seems plausible the correct update size for a founder running a failed startup is a couple percentage points of confidence in them along certain metrics.
I think MetaMed seems like more of an update than that, my basic reasoning being: 1) I think it was entirely possible to see what was wrong with the idea before they kicked it up, 2) accounting for the possibility of bad faith, 3) Constantin’s mortem suggests some maybe serious issues 4) I consider Zvi’s post-mortem to be more deflective than an attempt at real self-evaluation. So maybe like 6-9 points?
I uh, I don’t actually think Balsa is at all likely to be bad or anything. Please don’t let that be your takeaway here. I expect them to write some interesting papers, take a few minutely useful actions, and then pack it in [65%]. There’s no justification why these posts have been as long as they are except that I personally find the topic interesting and want to speak my mind.
I expect I got some things wrong here, feel free to let me know what errors you notice.
I think a lot of your individual arguments are incorrect (e..g. $5000 is a steal for MetaMed’s product if they delivered what they promised. This includes promising only a 10% chance of success, if the problems are big enough).
I nonetheless agree with you that one should update downward on the chance of Balsa’s success due to the gestalt of information that has come out on Zvi and MetaMed (e.g. Zvi saying MetaMed was a definitive test of whether people cared about health or signaling care, while Sarah lays out a bunch of prosaic problems).
I think “we” is a bad framing as long as the project isn’t asking for small donor funding.
I do think grand vague plans with insufficient specifics (aka “goals”) are overrewarded on LW.
OTOH I have a (less) grand vague project that I’m referring to in other posts but not laying out in totality in its own post, specifically because of this, and I think that might be leaving value on the table in the form of lost feedback and potential collaborators. A way for me to lay out grand vague plans as “here’s what I’m working on”, but without making status claims that would need to be debunked, would be very useful.
OTTH it’s maybe fine or even good if I have to produce three object-level blog posts before I can lay out the grand vague goal.
But also it’s bad to discourage grand goals just because they haven’t reached the plan stage yet.
I don’t think it’s a ding on his track record.
He tried a product.
It worked shockingly well for him.
He recommended others use that product.
This is a basic prosocial act. You haven’t made an argument that the product was low-quality, the failure of the company only shows that there wasn’t enough of a market for that particular product to sustain the company. For the most part I’m glad Eliezer advertised it while I could still buy it, it seems like the product was pretty great (though very expensive).
For context, here is an advert he gave for them, which has no endorsement of funding the company, nor says the organization is well run, and entirely focuses on his experience of the product (with the exception of one parenthetical).
Note: I didn’t read the HPMOR advert, I read the one here on LW which is different. It starts like this:
You’re right that he doesn’t make any specific verifiable claims so much as be very glowing and excited. It does still make me less inclined to trust his predictive ability (or trust him, depending on how much is him believing in that stuff vs building up hype for whatever reason.)
I do think this ad doesn’t line up with what you said re: “[...] nor says the organization is well run, and entirely focuses on his experience of the product (with the exception of one parenthetical).”
As I understand it, you’re updating against his recommendations of a product by his friends being strong evidence that the company won’t later go out of business. This seems fine to me.
I’m saying that his endorsement of the product seems eminently reasonable to me, that it was indeed life-changing for him on a level that very few products ever are, and that in general with that kind of information about a product, I don’t think he made any errors of judgment, and acted pro-socially.
I will continue to take his product advice strongly, but I will not expect that just because a company is run by rationalists or that Eliezer endorses the product, that this is especially strong evidence that they will succeed on the business fundamentals.
I think you were mistaken to call it a “ding on his track record” because he did not endorse investing in the company, he endorsed using the product, and this seems like the right epistemic state to me. From the evidence I have about MetaMed, I would really want to have access to their product.
As an example, if he’d written a post called “Great Investment Opportunity: MetaMed” this would be a ding on his track record. Instead he wrote a post called “MetaMed: Evidence-Based Healthcare”, and this seems accurate and to be a positive sign about his track record of product-recommendations.
Unrelatedly but from the same advert. I had not realized it was that expensive—this rings some alarm bells for me but maybe it is fine, it is in fact a medical service. I have been waffling back and forth and will conclude I don’t know enough of the details.
Regardless, the alarm bells still made me want to survey the comments and see if anyone else was alarmed. Summaries of the comments by top level:
> The words “evidence-based medicine” seems to imply “non evidence-based medicine”
> Will MetaMed make its research freely available?
> Proposals re: the idea that MetaMed might not improve the world save for their clients
> You should disclose that MIRI shares sponsors with MetaMed, detail question
> Please send this to the front page!
> I’m overall not impressed, here are a couple criticisms, what does MetaMed have over uptodate.com in terms of comparative advantage? (Nice going user EHeller, have some Bayes points.)
> Discussion of doctors and their understanding of probability
> MetaMed has gone out of business (3 years later)
> Is MetaMed a continuation of a vanished company called Personalized Medicine?
> A friend of mine has terrible fibromyalgia and would pay 5k for relief but not for a literature search of unknown benefit. I guess she’s not the target audience? (long thread, MetaMed research is cited, EHeller again disputes its value compared to less expensive sources)
> An aside on rat poison
> How might MetaMed and IBM Watson compare and contrast?
> Error in advert: Jaan Tallinn is not the CEO but chairman, Zvi is the CEO.
> Is MetaMed LW-y enough that we should precommit to updating by prespecified amounts on the effectiveness of LW rationality in response to its successes and failures?
There I will cut off because the last commentor is after my own heart. Gwern responds by saying:
And that is correct. But you don’t have to make a single prediction, success/fail, you should be able to come up with predictions about your company that you can put higher numbers on and we can see how those empirically turn out. Or you could even keep track of all the startups launched by prominent LW members.
In contrast, Michael Vassar (who was also on the project) says,
Which I don’t think anyone followed through on, perhaps because they then agreed with gwern?
Anyway—it seems plausible the correct update size for a founder running a failed startup is a couple percentage points of confidence in them along certain metrics.
I think MetaMed seems like more of an update than that, my basic reasoning being: 1) I think it was entirely possible to see what was wrong with the idea before they kicked it up, 2) accounting for the possibility of bad faith, 3) Constantin’s mortem suggests some maybe serious issues 4) I consider Zvi’s post-mortem to be more deflective than an attempt at real self-evaluation. So maybe like 6-9 points?
I uh, I don’t actually think Balsa is at all likely to be bad or anything. Please don’t let that be your takeaway here. I expect them to write some interesting papers, take a few minutely useful actions, and then pack it in [65%]. There’s no justification why these posts have been as long as they are except that I personally find the topic interesting and want to speak my mind.
I expect I got some things wrong here, feel free to let me know what errors you notice.
I’m torn because:
I think a lot of your individual arguments are incorrect (e..g. $5000 is a steal for MetaMed’s product if they delivered what they promised. This includes promising only a 10% chance of success, if the problems are big enough).
I nonetheless agree with you that one should update downward on the chance of Balsa’s success due to the gestalt of information that has come out on Zvi and MetaMed (e.g. Zvi saying MetaMed was a definitive test of whether people cared about health or signaling care, while Sarah lays out a bunch of prosaic problems).
I think “we” is a bad framing as long as the project isn’t asking for small donor funding.
I do think grand vague plans with insufficient specifics (aka “goals”) are overrewarded on LW.
OTOH I have a (less) grand vague project that I’m referring to in other posts but not laying out in totality in its own post, specifically because of this, and I think that might be leaving value on the table in the form of lost feedback and potential collaborators. A way for me to lay out grand vague plans as “here’s what I’m working on”, but without making status claims that would need to be debunked, would be very useful.
OTTH it’s maybe fine or even good if I have to produce three object-level blog posts before I can lay out the grand vague goal.
But also it’s bad to discourage grand goals just because they haven’t reached the plan stage yet.