I’ve been watching with interest the debates around how good minicamp was. I think we need to distinguish between two hypotheses:
A) Minicamp was well-run, the participants enjoyed it and subjectively estimated it was helpful, and it made everyone involved much more enthusiastic about rationality and motivated to pursue positive self-change.
B) Minicamp had objective effects on measurable rationality parameters like calibration, and objective long-term effects on things like lifetime success, tendency to help rationality-related causes, and ability of participants to enjoy their life.
Everyone who’s talking about how obvious it is that minicamp was a stupendous success is talking about A, and everyone who’s saying they’re not convinced is talking about B.
Most self-help doesn’t work—so with zero background information about a camp, our prior probability of B is low. That this is a rationalist camp is extra information: P(B|Rationality_Camp) is greater than P(B) alone if we believe rationality is a more effective self-help strategy than average. But this just brings us to where we were before the camp started; to say the evidence in the post above increases that estimate we’ve got to investigate P (B|A) - the probability that, given a camp gets glowing reviews and everyone loved it and thinks it changed their life, the it really is effective.
But A is a common feature of almost all self-help camps—googling “Christian retreat testimonial” canbeveryenlightening (add the phrase ‘changed my life’ to the query for best results). I think most rationalists would be very skeptical of most of the camps that manage to get such glowing reviews from their participants. So P(B|A) - the probability that data shows a real long-term effect given that everyone loved it and is wildly enthusiastic—may not be much higher than P(B).
So if you’re trying to prove A—that the camp was successful and everyone loved it and felt very motivated—you’ve more than succeeded by now. If you’re trying to prove B, keeping on giving more and more evidence for A isn’t really the way to go.
A better suggestion might be to tell people “We have strong evidence you’ll love the camp and feel transformed and enlightened, and we have some evidence that it will help because rationality is teachable and we’re trying to gather more specific evidence as the program continues.”
But A is a common feature of almost all self-help camps—googling “Christian retreat testimonial” can be very enlightening (add the phrase ‘changed my life’ to the query for best results). I think most rationalists would be very skeptical of most of the camps that manage to get such glowing reviews from their participants. So P(B|A) - the probability that data shows a real long-term effect given that everyone loved it and is wildly enthusiastic—may not be much higher than P(B).
I suspect that even Christian retreats do cause B. Just being around people with high expectations of you will cause you to rise to the challenge. This is important since simply measuring B won’t tell us whether the camp succeeded.
Everyone who’s talking about how obvious it is that minicamp was a stupendous success is talking about A, and everyone who’s saying they’re not convinced is talking about B.
That’s a really good point, and clarifies the issue enormously.
To be clear, I’m saying that minicamp had more of what you call B-type effect on me (so far) that many other such events. So I’m talking about B, not just A. From the OP:
Note that mini-camp was far from the first time I’ve travelled to an event to surround myself with like-minded peers working toward common goals. [...] I’ve been to many such workshops, inside and outside academia (~3 per year for the past 10 years). [...] Yet mini-camp is still topping my charts.
In particular, I’m saying that in my experience it was much more effective, B-wise, than the base-rate of generic peer gatherings like Christianity camps (which I’ve been to).
So no, not everyone who’s excited about minicamp is just talking about A. But yes, I agree with you, A is a lot of the conversation. I’m trying to focus on B.
everyone who’s saying they’re not convinced is talking about B.
Not everyone. The single highest-voted comment on the subject asks for “any attempt to extend and replicate this success” and “If it actually were a failure, how would we know? Would anyone there even admit it, or prefer to avoid making its leaders look bad?” While Academician’s subjective assessment adds nothing to Anna Salamon’s survey, the comment about hiring Luke actually addresses these complaints.
I’ve been watching with interest the debates around how good minicamp was. I think we need to distinguish between two hypotheses:
A) Minicamp was well-run, the participants enjoyed it and subjectively estimated it was helpful, and it made everyone involved much more enthusiastic about rationality and motivated to pursue positive self-change.
B) Minicamp had objective effects on measurable rationality parameters like calibration, and objective long-term effects on things like lifetime success, tendency to help rationality-related causes, and ability of participants to enjoy their life.
Everyone who’s talking about how obvious it is that minicamp was a stupendous success is talking about A, and everyone who’s saying they’re not convinced is talking about B.
Most self-help doesn’t work—so with zero background information about a camp, our prior probability of B is low. That this is a rationalist camp is extra information: P(B|Rationality_Camp) is greater than P(B) alone if we believe rationality is a more effective self-help strategy than average. But this just brings us to where we were before the camp started; to say the evidence in the post above increases that estimate we’ve got to investigate P (B|A) - the probability that, given a camp gets glowing reviews and everyone loved it and thinks it changed their life, the it really is effective.
But A is a common feature of almost all self-help camps—googling “Christian retreat testimonial” can be very enlightening (add the phrase ‘changed my life’ to the query for best results). I think most rationalists would be very skeptical of most of the camps that manage to get such glowing reviews from their participants. So P(B|A) - the probability that data shows a real long-term effect given that everyone loved it and is wildly enthusiastic—may not be much higher than P(B).
So if you’re trying to prove A—that the camp was successful and everyone loved it and felt very motivated—you’ve more than succeeded by now. If you’re trying to prove B, keeping on giving more and more evidence for A isn’t really the way to go.
A better suggestion might be to tell people “We have strong evidence you’ll love the camp and feel transformed and enlightened, and we have some evidence that it will help because rationality is teachable and we’re trying to gather more specific evidence as the program continues.”
I suspect that even Christian retreats do cause B. Just being around people with high expectations of you will cause you to rise to the challenge. This is important since simply measuring B won’t tell us whether the camp succeeded.
That’s a really good point, and clarifies the issue enormously.
Yup. I wish the conversation had started this way.
To be clear, I’m saying that minicamp had more of what you call B-type effect on me (so far) that many other such events. So I’m talking about B, not just A. From the OP:
In particular, I’m saying that in my experience it was much more effective, B-wise, than the base-rate of generic peer gatherings like Christianity camps (which I’ve been to).
So no, not everyone who’s excited about minicamp is just talking about A. But yes, I agree with you, A is a lot of the conversation. I’m trying to focus on B.
Not everyone. The single highest-voted comment on the subject asks for “any attempt to extend and replicate this success” and “If it actually were a failure, how would we know? Would anyone there even admit it, or prefer to avoid making its leaders look bad?” While Academician’s subjective assessment adds nothing to Anna Salamon’s survey, the comment about hiring Luke actually addresses these complaints.
No, he was also talking about B (link).